Designing and Implementing IDE Cross-Compatibility for Saros

worked on by: Tobias Bouschen
due by: 2020-06-30 (CW 27) Moved back due to Corona Pandemic (FU announcements)
due by: 2020-11-09 (CW 46)
initial presentation slides: PDF

Outline

Saros is a plugin designed for distributed pair programming*. Its goal is to provide the technical infrastructure for developers to concurrently collaborate on shared resources without being co-located. Furthermore, it provides the benefit of allowing the users to work in their usual work environments (IDEs) using their own configuration, thereby potentially increasing the productivity of pair programming sessions.

Saros was initially only designed for Eclipse, but later on was adjusted to also be available for other IDEs as well (mainly IntelliJ, but support for other IDEs would also be possible and was/is being worked on by other theses). The introduction of additional implementations for other IDEs broadened the usability and potential user base of Saros but also led to a new difficulty: Saros implementations for different IDEs were not cross-compatible.

Providing such cross-IDE compatibility would further increase the benefit of allowing the users to work in their usual environment by bridging the gap between different IDE users, thereby furthermore broadening the potential user base and use cases for Saros.

Pair Programming Footnote

If you are not familiar with pair programming, you can read this recent article from Martin Fowler about it.

Goal of the Thesis

The goal of this thesis is providing IDE cross-compatibility capabilities for the Saros plugin. To make it more approachable, I split it up into the following sub-goals:

  1. Analyze the current state of the Saros plugin and its IDE implementations.
  2. Identify potential roadblocks on the way to supporting IDE cross-compatibility.
  3. Resolve the necessary roadblocks.

Previous theses already worked on the related topic of extracting the core business logic into a separate package, the Saros Core:

Furthermore, this work was also continued through non-thesis-work by multiple Saros contributors over the years.

The process of creating a better IDE abstraction was also started more recently with the master's thesis "Saros: Refactoring the Filesystem" by Oliver Hanßen (2019) (written in German), which worked on providing a basic design for making the filesystem implementation IDE independent by moving to a reference point system.

As an almost all-encompassing work on Saros, a general overview of the field of agile development and pair/party programming as well as Saros, its development history, how it compares to other, related tools, and a detailed analysis of the design of Saros measured under the viewpoint of usability in an industrial setting is given in the dissertation "Industrially Usable Distributed Pair Programming" by Julia Schenk (2018).

Issues With Cross-IDE Compatibility

Saros was initially designed solely as an Eclipse plugin. As other IDE implementations were never planed, the interface and software architecture design was heavily based on Eclipse models to make the integration as easy as possible.

Later on, the idea of providing Saros implementations for other IDEs gained traction, leading to substantial efforts being made to extract any IDE-independent Saros business logic into a separate component, the Saros Core. But the basic design and architecture of the logic was not adjusted, leading to a situation where the Saros Core components still contain many assumptions of how different IDE concepts work that are heavily based on the Eclipse model.

As a result, implementing Saros for other IDEs is not as easy as it could be. Some aspects of the Core interfaces are still closely modeled after Eclipse counterparts, leading to issues when trying to implement them for other IDEs.

An important additional factor for this thesis is that I make sure that my knowledge of the Saros/I implementation and the used IntelliJ APIs does not influence the design process in a similar manner. Otherwise, the resulting architecture could suffer from the same issues as the current one.

Milestones and Planning

# Past Date CW Goals
1 28.01.20 CW05 A basic overview over the Saros architecture and possible roadblocks has been achieved
2 21.02.20 CW08 A list of steps to take on the way to implementing IDE cross-compatibility has been compiled
3 27.06.20 CW26 The Saros filesystem interface has been revamped
4 27.06.20 CW26 Basic IDE cross-compatibility has been implemented

The list of steps to take on the way to supporting IDE cross-compatibility is quite extensive. The milestone of having refactored the filesystem handling is quite arbitrary and was chosen before the list of steps was defined. However, as the refactoring is probably the largest single task, it still offers a good milestone to compare progress against. The list of milestones might still be adjusted during the process of working on the defined steps.

Milestone 1 – Roadblocks

To identify roadblocks, I had a general look through the existing packages and classes in the Saros Core, Saros/E and Saros/I. Furthermore, I encountered some of these issues while working on Saros/I.

Initially, I have identified the following roadblocks:

  1. The Handling of Editor Contents
  2. The Handling of Modules/Project
  3. Determining Which Resources to Share

Roadblock 1 – The Handling of Editor Contents

With the current operating systems, there are two different ways of signaling the ending of a line in a text file: \n for UNIX-based operating systems and \r\n for the Windows operating system.

In Eclipse, like in most IDEs and editors, the line ending written in the file are also used when processing it internally or representing it in the editor. In IntelliJ, however, line endings are normalized to UNIX line endings internally. All line endings in the file are substituted with the UNIX line ending when it is read from disk and the original line endings are then re-substituted when the content is written back to disk.

This poses a big issue for IDE cross-compatibility as, even when working on the same content on disk, the document content available in the IDE does not have to match. In particular, when trying to maintain a session between an IntelliJ instance and an Eclipse instance running on Windows, Saros will always detect an inconsistency in the shared files as the line endings will not match. Furthermore, as the two line separators are not of equal length, the UNIX line separator consisting of one character and the Windows line separator consisting of two characters, the character-based offsets will be different in the two files. This poses another challenge for Saros, which relies heavily on such offsets to apply remote changes correctly and display awareness information such as remote cursor locations and selections.

Roadblock 2 – Issues With the Current Project/Module Handling

Different IDEs use different models to represent the structure of the aggregates of files representing a piece of software. As Saros wants to interact with these files, it needs to be able to interface with this logic.

Since Saros was first only designed for Eclipse, the interface to interact with the filesystem logic matches the Eclipse counterpart, going so far as to simply refer to the Eclipse documentation if no other explicit documentation was given for a method or class. The design was also not amended when extracting the logic to the Saros Core, meaning it was still very much based on the Eclipse counterpart. This makes the implementation of Saros for other IDEs more cumbersome as the IDE-specific filesystem interface has to be somehow mapped onto the Saros (or rather Eclipse) filesystem interface. Even when such a mapping is possible, the underlying implementation still has to be IDE specific to match the IDE filesystem model. This can lead to a confusing mix of names and concepts in the implementation, especially when naming schemes clash in the different IDE concepts, an example being the usage of the term "project" in the Eclipse and IntelliJ model.

Though this issue with mapping new IDE implementations onto the Core interface is cumbersome when implementing Saros for new IDEs, it does not provide as much of an issue for IDE cross compatibility. A more pressing issue is that the current sharing logic uses IDE-specific concepts to determine the files to share with other participants. Saros for Eclipse allows the user to share projects while Saros for IntelliJ allows the user to share modules. While these concepts are generally comparable in scope, it is not always possible to create a clean mapping between Eclipse projects and IntelliJ modules for the same piece of software as the filesystem models of Eclipse and IntelliJ are far to different. This especially becomes an issue when trying to create a project/module as part of a cross-IDE Saros session as the other IDE models won't necessarily provide all the information necessary to correctly represent it in the local model. These complications grow with the complexity of the shared project/module setup.

As an example, even with a relatively normal setup, like the Saros project itself, a mapping between Eclipse projects and IntelliJ modules is far from trivial: Eclipse sees each Saros component (e.g. Core, Eclipse, IntelliJ) as one project. IntelliJ, on the other hand, imports every Saros component through Gradle as three modules, each containing different parts of the tree: a base module containing the general resource tree, a "main" sub-module containing only the source roots and resources, and a "test" sub-module containing only the test roots and resources. As a result, there is no clean mapping between Eclipse and IntelliJ as the same resources that are represented by one project in Eclipse are spread across three modules in IntelliJ. Furthermore, some of these IntelliJ modules can't be represented as a unified tree (without including parents outside the module) as they are the combination of different resource trees. There is no way of representing such a collection of resources as a single project in Eclipse.

Roadblock 3 – Determining Which Resources to Share

Generally, projects/modules contain resources that should not be shared through Saros, examples being automatically generated artifacts or files containing user specific data. A good definition for most of these files would be "files that should not be kept under versioning", i.e. that should be ignored by the used version control system (e.g. Git, Subversion, etc.).

The current approach of Saros is to use local IDE mechanisms to determine which resources not to share. Most IDEs offer the option to ignore resources like build artifacts in the local workspace in order to exclude them from functionalities like inspections, searches, and refactoring tools. Eclipse offers the concept of derived resources and IntelliJ the concept of excluded resources for this purpose.

The issue with this approach is that, in both cases, these concepts are generally only used to mark resources that are generated as part of the compilation process. However, there also are other automatically generated resources and resources that contain user specific data that should not be shared between Saros instances. When implementing Saros to only work for one specific IDE, it is quite simple to cover most of these special cases with dedicated logic to exclude such resources as most of them can be queried from the IDE. But this only works if the set of non-shared resources determined through the IDE specification is the same for all participants. If participants use different IDEs, the set of such additional non-shared resources might vary between the participants, leading to a perceived desynchronization as Saros detects files for some participants while they are ignored for others.

Milestone 2 – Steps

Initial, I created a list of steps necessary in order to reach IDE cross-compatibility. These steps offered a good jumping-off point but were to vague to be solely relied on during the implementation phase. Instead, I proceeded in an agile fashion, deciding on what to do next in an ad hoc fashion.

A good overview of the implementation process is given in the work log.


Work Log

Preparation

Activities

I also looked for general articles about the usability of (distributed) pair programming tools but was not able to find many sources that brought new information to the table. Most of the important information is already given in the dissertation from Julia Schenk (which also references the sources it uses).

Week 00 + 01 (CW 03+04)

currently working on milestone 1

Activities

Results

Next Steps

Problems

Week 02 (CW 05)

currently working on milestone 1 and 2

Activities

Results

Next Steps

Problems

Week 03 (CW 06)

*currently working on milestone 2

Activities

Results

Next Steps

Problems

Week 04 (CW 07)

currently working on milestone 2

Activities

Results

Next Steps

Problems

Week 05 (CW 08)

currently working on milestone 2

Activities

Results

Next Steps

Problems

Week 06 (CW 09)

currently working on milestone 2 & 3

Activities

Results

Next Steps

Problems

Week 07 (CW 10)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 08 (CW 11)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 09 (CW 12)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 10 (CW 13)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 11 (CW 14)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 12 (CW 15)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 13 (CW 16)

currently working on milestone 3 (now actually)

Activities

Results

Next Steps

Problems

Week 13 (CW 17)

currently working on milestone 3 (now actually)

Activities

Results

Next Steps

Problems

Week 14 (CW 18)

currently working on milestone 3 (now actually)

Activities

Results

Next Steps

Problems

Week 15 (CW 19)

currently working on milestone 3 (now actually)

Activities

Results

Next Steps

Problems

Week 16 (CW 20)

currently working on milestone 3 (now actually)

Activities

Results

Next Steps

Problems

Week 17 (CW 21)

currently working on milestone 3 (now actually)

Activities

Results

Next Steps

Problems

Week 18 (CW 22)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 19 (CW 23)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 20 (CW 24)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 21 (CW 25)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 22 (CW 26)

currently working on milestone 3

Activities

Results

Next Steps

Problems

Week 23 (CW 27)

currently working on writing

Activities

Results

Next Steps

Problems

Week 24 (CW 28)

currently working on writing

Activities

Results

Next Steps

Problems