You are here: Wiki>SE Web>ThesesHome>ThesisDPPXIII (15 Feb 2010, StephanLau)Edit

Verbesserte Präsenz durch Screensharing für ein Werkzeug zur verteilten Paarprogrammierung (ThesisDPPXIII)

Improved presence through screen sharing for a distributed pair programming tool (Bachelorarbeit)

von Stephan Lau, betreut von Karl Beecher und Stephan Salinger

Antrittsvortrag / Preparation presentation

is here

Ziele

Saubere und schnelle Implementierung von Screensharing in Saros wink

Planung / Approximate timeline

  • registration: 17.11.2009
  • delivery: 9.2.2010
  • ill: 21.12.09 - 8.1.10 (19 days)

  1. Week 1-2 9. - 20.11
    • base architecture and requirements for screensharing
    • start to implement
  2. Week 3-4 23.11 - 4.12
    • start integration in saros
      • UI
      • bytestream via XMPP
    • only passive features first
  3. Week 5-6 7.12 - 18.12
    • additional features
      • active screensharing
  4. Week 7-8 21.12 - 1.1.2010
    • prepare feature for release
    • alternative encoders
    • test/optimize
  5. Week 9-10 4. - 15.1
    • deal with scientific questions, theoretical preparation for thesis
      • evaluate screensharing
        • what has been done?
        • how is screensharing accepted (by saros' users)?
        • how (or does) screensharing improves saros?
        • are there remaining problems?
      • screensharing <-> collaboration awareness
        • how are they related?
        • can they be combined?
  6. Week 11-12 18.1 - 5.2.
      • final work on thesis

Anforderungen / Requirements

  • functional
    • client (consumer)
      1. see screen (or part) of remote desktop
      2. zoom into desktop
        • useful when desktop has higher resolution than video
      3. click and/or point on the remote desktop
        • send this activities to server
      4. resize video to size of player
    • server/host (provider)
      1. provide local desktop
        • capture desktop, put it in videostream
      2. execute actions send by client
      3. pretend overload of connection between server and client
        • control bitrate of stream and adjust when necessary
  • non-functional
    1. remote desktop is viewed in realtime or as fast as possible
    2. source of transmitted images and used encoder is replacable (by architecture)
    3. effective encoding
      • use xuggler
    4. fallback when native encoder not installed

Optional

Will be implemented when time is left.

  • focus sharing on one or more windows
  • support multi-monitor environments
  • switch between screen and webcam
  • record session, maybe with activities
  • share some parts of eclipse only
    • e.g. a console or something
  • one host, many clients
  • don't use slow codecs (e.g. libx264) on slow machines
    • ability to detect this

Anwendungsfälle / Use Cases

see ThesisDPPXIIIUseCases

Implementierung

see ThesisDPPXIIIProgramming

Literatur

see ThesisDPPXIIILiteratur

Status

StreamServices

This layer adds the ability to define services/features which need pure streams (Java's InputStream / OutputStream).

Current state

running, but sending is not reliable enough

Working features

  • session negotiation
  • smooth termination of sessions
  • configure characteristics of streams to meet non-functional requirements
    • size of buffer
    • minimal size of chunks transferred
    • maximal delay between sending sending data

Problems

  • =BinaryChannel= deadlocks sometimes -> data lost, stream broken
  • when an OutputStream is closed which still contains data they will be lost
    • reason: MetaPacket send immediately, but data is queued

Concrete TODO's

  • solve stream-close problem (s.a.)
  • review errorhandling
  • =StreamSessionOutputStream= : public methods to get buffer's state
  • =SteramSessionInputSession= : count read bytes, public available

Nice-to-have features

  • integrate SubMonitor into streams methods
    • not really necessary, but DataTransferManager uses it

ScreenSharing (or VideoSharing)

This is the main feature of this thesis. It's purpose is to see the desktop of a participant in a saros-session. The implementation is a general framework to supply any sequence of images from one to another saros-user.

Current state

basically runs, a session can be started

Working features

  • start a session, see remote desktop
  • capture mouse- and keyboardevents in player
    • switch between fullscreen and mouse following
  • set up basic preferences via eclipses preferences

Problems

  • xuggler
    • resampling of images takes a lot of computation (more than compressing the picture) because it's not accelarated by hardware
    • on mac only supported for 64-bit java 1.6
      • good argument to implement different en-/decoder wink
  • player
    • draw and keep initial background
    • scrollable or resize input
  • desktop/screen
    • zoom-in/-out when following mouse
  • preferences
    • validate new chosen values (e.g. invalid container for xuggler)

Concrete TODO's

  • implement pure java de-/encoder
    • promising possibilities
      • screenshare demo in Smack org.jivesoftware.smackx.jingle.mediaimpl.sshare used this as statrting point
      • Dayon!
        • hard to understand, poorly commented
        • seems that pictures are only gray
      • Java Remote Control
        • complex structure, is based on XAPP
        • poorly commented
  • monitor and throttle used bandwidth
  • make GUI usable
    • good icons for view and starting session
    • fix mentioned problems in player
  • preferences
    • switch resolutions via combobox
    • choose different en-/decoder
  • stop and dispose session
  • errorhandling

Nice-to-have features

Not sure I can make it for this thesis.

  • execute click/type action on host (active screensharing)
    • user-rights management in preferences
  • record session (transmitted stream and activities)

Berichte

Vorbereitungsphase / Preparatory stage

Week ( 1.10 - 2.10.09)

Activities
  • took part in testing of release 9.10.2
  • search existing libraries/programms in java for screensharing or screencapture (on javaforge, berlios, freshmeat, sourgeforge)

Results
  • invitationprocess seems to be buggy -> not released yet

Problems
  • not found many existing tools.. am I the first one using this in java!?!?

Comments

Goals
  • get an overview to screensharing
  • find some more javastuff concerning this topic

Week ( 5.10 - 9.10.09)

Activities

Results
  • seems that I have build screenshare on my own
    • use a simple videostream for transmitting the screen
  • did not found other existing screensharingtools or apis for dynamic streaming of content
  • chosen an api for videoencoding Xuggler
    • based on ffmpeg
      • wide range of supported codes
      • active development
      • fast encoding due to native code
    • easy to install on windows, mac, linux with integrated and compiled ffmpeg

Problems
  • videoencoding is very complex
    • choose a preferred container and videoformat
    • understand lots of possible settings an encoder uses

Comments

Goals

  • running prototype

Week ( 12.10 - 16.10.09)

Activities

  • explored xuggler

Results

  • found 'hidden' settings for encoders in xuggler

Problems

  • most settings applied not to codec h.264

Comments

  • was ill from 13.10 to 16.10

Goals

  • see last week

Week ( 19.10 - 23.10.09)

Activities

  • worked on prototype
  • set up libx264 encoder to use cbr (constant bitrate encoding)
  • meet the given bitrate and don't exceed it
  • first checkin of prototype into svn

Results

  • prototype
    • client-server program, streaming via network
    • dynamic resizing of inputframes to encoder
    • console in server to change and view current enviroment
    • change fps while streaming
    • use vbv-buffer (in ffmpeg) to guarantee the bitratelimit (in some cases, esp. low bitrates, without vbv some peaks appeared)

Problems

  • due to update of xuggler, 'libx264' needed other settings than default ones
    • presets by ffmpeg did not work
    • examples by xuggler used not cbr-mode
    • bruteforce trying to find appropriate settings
  • in cbr mode x264 has a noticable delay
    • seems that encoder collects all frames between two keyframes, encodes them at once to keep given bitrate and finally pushes them into videostream

Comments

  • xuggler updated to 3.3 on 21.10, so problems on setting up 'libx264' disappeared (see last week)

Goals

  • release saros
  • make weekly reports
  • prototype
    • deal with temporary low bandwidth

Week ( 26.10 - 30.10.09)

Activities

  • improved prototype
  • releasemanager in this cycle
  • tested new release

Results

  • prototype
    • extended console
    • threaded server
    • change resolution while streaming -> problem
    • introduced and second stream for controlling the transmission
      • client sends statistics about delay etc. to server
  • releasemanager
    • collected all changes since last release
    • prepared branch for testing

Problems

  • resizing the livestream is not possible
    • used container (flv) does not support more than one videostream
    • other containerformats don't seem to work frown, sad smile
      • either not streamable (mpeg4, ...)
      • or don't work w. xuggler? (mkv, ogg, ...)
  • setting a new framerate on the fly causes a big delay on client
    • reason can't be figured out, maybe is the old timebase (based on old framerate) the reason
    • try to drop frames instead of changing framerate

Comments

Goals

  • prepare the presentation for thursday next week
    • plan advance for the next weeks
    • give an overview about screensharing
    • details of planned implementation
      • features
      • requirements
      • integration into saros
  • find reason why some containerformats don't work in xuggler
  • prototype
    • deal with temporary low bandwidth

Working stage

Week ( 2.11 - 6.11.09)

Activities

Results

Problems

Comments

  • not done
    • find reason why some containerformats don't work in xuggler

Goals

  • complete base architecture for screenshareapp
  • send some bugreports to xuggler

Week( 9.11 - 13.11.09)

Activities

  • worked on base architecture
  • meeting with Stephan and Karl on wednesday

Results

  • wrote usecases (ThesisDPPXIIIUseCases) and requirements
  • developed modules (functionality, responsibilities)
  • meeting
    • shorten iterations to 2 weeks
    • immediately integrate screensharing into saros
  • some bugreports for xuggler

Problems

Comments

  • this week was not really productive, i increase it!

Goals

  • adapt plan due to meeting
  • finish architecture
  • start integration in saros

Week ( 16.11 - 20.11.09)

Activities

  • preparation for implementation (requirements engineering)
  • worked on implementation
  • adapted plan

Results
  • architecture and modules, see ThesisDPPXIIIProgramming
  • started to code (see my repository)
    • ported prototype to new architecture
    • local testing via piped streams possible

Problems
  • still some open questions for architecture
    • who controls when a frame will be rendered, encoder or connectionmanager?
    • what is needed at clients side to initiate and handle connections?
      • need some details concerning saros, should be done this week
    • how can i handle configurations?

Comments

Goals
  • integrate into saros
    • view for clients player
    • bytestreams for connection between two users

Week( 23.11 - 27.11.09)

Activities
  • programmed feature
    • startup with saros
    • get a network stream

Results
  • programming
    • hooked into picocotainer
    • tried to get network stream
  • entered tests 3.2 - 3.7 in testlink

Problems
  • is not easy to get network stream
    • find a way to get one via jingle, but this seems not to be accepted -> meeting tuesday

Comments

Goals
  • find solution for network architecture (meeting tuesday)
    • let saros be packetbased, wrap a stream with them
    • get a real stream via jingle/socks
    • not completely my part, sandor and henning have to help me
  • start ui
    • player
    • session initiation, negotiation

Week( 30.11 - 4.12.09)

Activities
  • worked on streaming with saros
  • assisted releasemanger
  • tested saros for release

Results
  • developed framework for streambased services
    • simple filetransfer-test worked

Problems

Comments

Goals
  • test streaming with video, finish it completely
  • start ui stuff

Week( 7.12 - 11.12.09)

Activities
  • worked at our new streaming interface
  • played a bit with swt and eclipse views
  • implemented file sending based on streaming

Results
  • streaming: improved it
    • performance (chunkSize and maximalDelay)
    • reliability, send packets same order they arrive
    • introduced soft stop
  • UI
    • created first view (canvas with simple statusbar)
    • discovered swt: widgets, layouts, ...

Problems
  • not sure whether streaming is now reliable enough... spend already a lot of time there, but sometimes strange errors occur
Comments

  • don't get so far to test videosharing with stream interface

Goals
  • integrate view into videosharing
  • create a StreamService for video, test it in saros

Week( 14.12 - 18.12.09)

Activities
  • worked on view for playing video
  • implemented example service for sending single files in a session
  • improved streaming
  • integration of videosharing-prototype into saros

Results
  • videosharing
    • created branch, moved classes
    • created simple view for showing video with canvas and statusbar
  • streaming
    • can add serialized objects to metapackets
    • developed improving concepts
      • streams need to be closed
      • output should block when input is not reading
  • file sending
    • show speed in task
    • cancellable

Problems
  • videoplayerview is swt, but images are awt: need to embed an awt canvas
  • sometimes streaming hangs without obvious reason, need to figure out

Comments

Goals

  • improve streaming (s.a.)
    • esp. robustness (errorhandling, disposing)
  • make videosharing work in saros
    • embed awt-canvas in player
    • create preferences (pages)
    • catch input in player's canvas, send them to host

Week( 21.12 - 25.12.09)

Ill

Week( 28.12.09 - 1.1.10)

Ill

Week( 4.1 - 8.1.10)

Ill

Week( 11.1 - 15.1.10)

Activities
  • finish screensharing and streaming for release
  • documented state of work in wiki
  • write paper about technical realization

Results
  • screensharing
    • disposing
    • fast awt-canvas for displaying video
    • catch input in videoplayer and send to host
    • pure java encoder taken from smack-demo
  • streaming
    • errorhandling/disposing
    • many fixes
  • paper - 50% done

Problems
  • alternative encoder is too slow, works only w. 1 FPS
    • try to optimize

Comments
  • screensharing not ready for release
  • streaming could be ready
    • christopher fixes binarychannel
    • must be reviewed (and afterwards fixed)
    • any feature uses it

Goals
  • make features ready for use for release next week
    • optimize other encoder
    • UI
      • players toolbar
      • resizing
      • preference dialogues
  • paper finished until wednesday

Week( 18.1 - 22.1.10)

Activities
  • writing mentioned paper
  • make screensharing ready for release
  • make streaming ready for release
  • reviewed some patches for release

Results
  • paper done
    • no feedback yet
  • screensharing works
    • UI minor bugs

Problems
  • seems that there's not enough to review both features
    • maybe not released or only experimental?
  • alternative encoder
    • really bad performance, only 1 fps at low resolution, even at fast machines

Comments
  • only core features in screensharing work

Goals
  • release a new version for Saros
  • finish streaming and sending file during session

Week( 25.1 - 29.1.10)

Activities
  • helped releasing and testing saros 10.1.29
  • worked on streaming and screensharing
Results
  • streaming
    • created patch for reviewing
  • screensharing
    • worked towards creating patch, not done yet

Problems

Comments

Goals
  • next weeks
    • get screensharing stable
    • focus on thesis
    • improve streaming according to reviews
  • next week
    • relax @ holiday in austria

Week( 1.2 - 5.2.10)

holiday in austria

Week( 8.2 - 12.2.10)

Activities
  • worked on thesis
  • improved streaming

Results
  • created outline
  • read and chose some papers, see ThesisDPPXIIILiteratur
  • streaming
    • almost done, thread-pools are missing

Problems

Comments

Goals
  • write parts about saros, pp, dpp and ca
  • create new patch for streaming to be reviewed

Week (15.2 - 19.2.10)

Activities
  • read some more papers
  • prepared streaming for release

Results
  • finished streaming patch, which will be reviewed for release

Problems

Comments
  • ill

Goals
  • get better

Week (22.2 - 26.2.10)

Activities
  • adapted patch for streaming

Results
  • streaming-patch
    • adapted due to hennings and flo's review

Problems
  • patch done too close to release, not released

Comments
  • still ill

Goals
  • get finally better
  • find some papers concerning video (esp. bandwidth-adaption)
  • write parts about saros, pp, dpp, ca and related work

Kommentare

Wer will, wer will, wer hat noch nicht? Ich würde mich über jede Anregung, Anmerkung, Kritik freuen. smile

 
Topic revision: r17 - 15 Feb 2010, StephanLau
 
  • Printable version of this topic (p) Printable version of this topic (p)