• Browse GIT
• View
• Edit
• Upload
• History
• Print

BundleSuggestions

A description and discussion on different solutions for how to do bundling.

Tips:

• You can use color, e.g. Bo uses this colour and Richard uses this colour when commenting in sections of which you are not the primary author.
• If you want to refer to posts in the LyX developers' list, it can be done as follows:
  Referring to a specific post: LyXDevelPost:107251
  Referring to a thread: LyXDevelThread:107251
  Referring to the root of a thread: LyXDevelThreadRoot:107251

1.  Your vote

1. Bo Peng: the implementation for my proposal can be made ready for 1.6.0.
2. Jurgen: this feature is too late for 1.6.0
3. Abdel: this feature is too late for 1.6.0
4. Pavel: there is no conclusion now so let us cool off this feature for a while
5. Andre: supports the first proposal.

2.  Richard Signs Off

When logging onto the wiki to do something suggested by Christian, I discovered that Bo has taken to using profanity. Mild profanity, yes, but profanity nonetheless. I am therefore terminating my involvement in this discussion. Whether I'll continue to be involved with LyX at all is an open question.

3.  Use cases

Bo: For user cases that make full use of the first approach, please refer to my answer to "What is you motivation for individual embedding?"

3.1  UC1: Create a compilable bundled document that can be unbundled later.

In this use case, users work mostly with external files because they are simpler to work with. Users would save his files to a repository. When he needs to send the document to a co-author who does not have access to his reposiroty, he would pack his files in a bundled format. His co-author would be able to open, view and edit it and send the document back. The user would then unbundle this file, restore the external files to their original locations, leaving more or less the original .lyx file, diff the changes and commit.

Please note that, in this case, you are going to want to bundle everything. So this use case is equally supported whether we have inset-specific embedding or not. And that, in my view, is the only real issue between the two approaches.

There are quite a few differences. By using 'global bundling', you introduce a 'bundle editing' mode and major change to .lyx files. Also, although bundling looks alike in both approaches, 'individual embedding' allows a much more friendly in-place unbundling feature. Individual embedding also allows the continuous use of external files.

3.2  UC2: Create a self-contained document template

In this use case the purpose is to create a self-contained document template, e.g. for thesis students.

Ideally the student should just be able to open the bundle and start writing on his thesis.

Notes: I'd imagine that such a page ought to contain a README in addition, explaining what's what... /Christian

We have lyx comment insets. I use them for such purposes regularly. /Bo

I really do not see a need for LyX itself to support this sort of thing. The document template can be distributed as a zip file, or whatever. So if this is what inset-specific embedding is all about, then, well, I think it's definitely not worth the cost in code complexity.

4.  Bo's suggestions (version 1.2, with a complete implementation)

The basic idea is that a file can be embedded into the .lyx document. Embedded files are invisible to users. They can have a link to its original external file, or be anonymous if such information is removed.

1. Each inset has its own embedding status. There is no global embedding flag.
2. Customized layout (.layout), class (.cls, .sty), bibtex style (.bst). and other files can be embedded. This allows the creation of a completely self-contained document.
3. The embedded files are written directly in the '.lyx' file encoded in base64.

Inset level features:

1. Inset context menu "Embedded" can be used to show the embedding status of an inset, and turn on/off embedding. This is a on/off switch for graphics, external and include inset, where unembed is only allowed when an external file is available and identical to the embedded version. Because a bibtex inset can have more than one embedded or external files, the context menu becomes 'Embed all bib files' and 'Unembed all bib files'.
2. Inset context menu "Save embedded file as". Save the embedded file to an external file.
3. Inset context menu "Update from external file". Update the embedded file from an external file.

Note: The external file used in operation 2 and 3 becomes the new 'external fle' of the embedded file. Un-embed can be then be used to make an inset refer to this external file.

Buffer level features:

1. "Document -> Embedded Files -> Embed all files". This will embed all embeddable files in a buffer.
2. "Document -> Embedded Files -> Make embedded files anonymous". Remove the original filenames of embedded files because the original name may contain sensible machine-specific information.
3. "Document -> Settings -> Embedded Files". Arbitrary files such as layout and class files can be embedded through this dialog.
4. "Document -> Embedded Files -> Un-embed files with external copies". This operation will un-embed all insets with an existing and identical external file. This feature can be used to in-place unbundle a file if only a few external files are changed in the embedded .lyx file.
5. "File -> Export -> LyX with extracted embedded files". This operation will create a new directory filename.ext_lyx under the document directory, extract all embedded files with known original names in a way that the original external file structure can be preserved. For example, filename.lyx may be a level down that directory with an inset referring to ../figures/figure.png. A Python script is used to allow such unbundling to be done outside of lyx.

Note: The export operation is not recommended because the exported .lyx file may not be compilable due to OS differences (e.g. if c:\blah is extracted as $DOC_DIR\filename.extracted\c\blah).

4.1  Expected workflow:

1. When a user needed to send a document to his co-author, he chose 'Document->Embedded Files->Embed all' and embed all the external files. He could optionally embed .cls, .layout, .bst files to make his document self-contained.
2. When a co-author received this document, he could view, edit and compile this document directly. When he finished editing the document, he could choose 'Embed all' to embed new external files.
3. The original author received the document and chose 'Document->Embedded Files->Unembed files with external copies'. All embedded files are unembedded except for the ones that have been modified or added by his co-author.
4. When an embedded file needs to be modified, a user can 'unembed' (or 'save as' then 'unembed' if there is no existing external file) to make this inset external, or 'update from external file' to update directly from an external file.
5. A user can extract all embedded files from a .lyx document using an export operation. Due to OS differences, the extracted file may not be compilable.
6. Optionally, a user can anonymise all embedded files to avoid saving machine-specific path information in the .lyx file. These embedded files will not be extracted during export.

4.2  Major benefits:

1. There is no change to existing .lyx files. The .lyx format is still in plain text.
2. Users can work with external, potentially out of tree, files as before.
3. Users can embed/un-embed individual inset, bundle/unbundle all external files, without loss of information.

4.3  The criticisms:

• These 'embed' checkboxes cluster the inset dialogs.
  Note that this is true all the time. Every user has to see these checkboxes, even if she has no intention of ever using embedding. This approach is therefore very "in your face". It alters the LyX UI in substantial ways, to support a feature that, for all we know, very few users would ever use. For note: In the first use case described above, you would (almost?) always want to embed everything. So the justification for individual embedding must be the second use case. But, as I said, it's not compelling.
  This is no longer the case when the implementation uses context menus.
• The .lyx format is more difficult to work with than a zip format (approach two). Base64 strings may confuses index-generators such as google desktop.
  Bo: The plain text file is more svn friendly, easier to work with for tasks other than extracting external files. I do not really care about index-generation applications.
  Richard: This isn't at issue between the approaches, since either one could use base64 and either one could use zip.
  Bo: There is no bundled mode in my approach so using a zip basically means turning the lyx format to binary. That is to say, my approach can not use zip. In your approach, you have a bundled mode which is filename.lyx + filename.lyxdir. LyX has its own compression mode. It would be a really bad idea to introduce another 'wrapped' mode for your approach so you would eventually use a zip format, namely, use compression to wrap your bundle.
  Note: Abdel thought that Base64 is impropriate for large embedded files. For what it's worth, Richard agrees.
• The code needed to support individual embedding is much more complex than if we do not support individual embedding.
  Bo: This is unconfirmed because global embedding is more intrusive, and may need a lot more code to iron out the problems.

5.  Richard's suggestion (version 1)

1. A 'bundled mode' is introduced. File format and working style are preserved in 'unbundled mode'.
2. In the bundled mode, all external files are copied to a subdirectory filename.lyxdir.
3. Switching from unbundled to bundled mode will copy all external files to this subdirectory.
4. Switching from bundled mode to unbundled mode will not copy any file. It merely stops copying files to this directory.
5. Original filenames are stored in a session file. The original user on the original machine can 'update from external' to update the embedded version from the external version. This update process may be automated in some way.
   Richard suggested that he can also store these links with .lyx file
6. If compression is turned on, filename.lyxdir will be zipped to filename.lyx.
   Richard suggested that he can also use base64, he has not decided on a file format to use. However, this is not at all the case, he needs to use zip. Note that JMarc, Abdel prefer a zip format.
7. When a compressed bundled file is opened, filename.lyxdir will be opened in the buffer temp directory (temporarily invisible to users), and be moved to the document directory when compression is turned off.

5.1  Major benefits:

1. No change to .lyx file format (but with permanent changes to the content of a .lyx file and all external files when switching to bundled mode, even if users are not editing anything. More about this later.)
2. The zip format is easier to work with when lyx is not available.

5.2  Criticisms:

• Having a 'document' being a file + a directory is unnatural. It causes confusion to users and some practical problems mentioned below.
• This method is more intrusive to how users work with lyx. They need to learn how to insert a file in this bundled mode, how to create a single-file bundle using compression, and how to handle this filename.lyxdir directory. In contrast, the first approach only introduces an 'embed a file' concept.
• Not friendly to existing files. A .lyx file will be permanently changed, with all external files copied, if it is turned to the bundled mode. Switching back to unbundled mode will not recover the original .lyx file.
  Richard: I'm offering "restore original file" as an additional option.
  Bo: That is better. If a user accidentally turns on the bundled mode, at least there is a (complicated) way to restore his files. Note that users of the first approach need only to "Un-embed files with external copies" to undo "Embed all". This feature can not be used by the second approach because it relies on individual embedding.
  Richard: I guess I don't understand why "Restore original file" is more complicated than "Un-embed files with external copies". Seems to me the functionality is the same.
  Bo: Using 'Restore original file', lyx would create a subdirectory with the extracted document. Users have to close lyx, examine and copy files back, and re-open the document. Using 'Un-embed files with external copies', untouched files are un-embedded in place, leaving changed or new embedded files embedded.
• Troubles with content management systems such as subversion. Other than having to reorganize existing files for existing documents because of 1), users have to guess which files are added to filename.lyxdir during editing to add them to subversion. This is because lyx determines where to put them, under which name. (This is troublesome to me, but maybe not to others)
  Richard suggested that the "world's simplest script" can handle this problem.
• Difficult to work with external files. In the bundled mode, all external files are bundled so users have to rely on (auto) update from external file to make their changes to external files available to lyx. Once a file is bundled, there is no way to unbundle it because there is no individual bundling. Switching to unbundle mode will not unbundle the file, this is potentially very confusing.
  There is no difference here. If files are embedded, you also have to rely upon update. And of course switching to unbundled mode leaves you with the original file in the bundle directory.
  Users can not use external file in the bundled mode. In Bo's approach, working with external files are continued to be allowed. Users can embed/un-embed at the inset level, without a global mode change.
• Can not maintain meaningful external file structure. External files are copied to filename.lyxdir in a predefined structure with possible aliased names to avoid name conflict. It is argued that users can keep their chapter1/figure1.png chapter2/blah structure for 'update from external' purposes, but this option is only meaningful to the original author. Also, once these external files are removed (they are not part of the document), the structure is lost.
  This is just wrong. The external--internal map is maintained here almost exactly as in the other proposal.
  Let me clarify. In my proposal, there is only one 'external file structure'. These files can be all embedded so there is no structure, or all external, or anything in between. The key point here is that there is one structure to care about. In Richard's proposal, there are 'external file structure', and 'internal file structure' within filename.lyxdir which is unstructured. Because Richard allows direct modification of files in the 'internal structure', that structure is actually also meaningful to users. This is conceptually very complicated.
• There are two visible copies of the same external file the external one and the bundled one. Richard has already offered 'updated from external' to update the bundled copy from the external copy. But for those users who would like to use the old modify-external-file way, they have to dig in this filename.lyxdir folder, locate the corresponding file (which can be renamed) and update it directly. This should be considered as hacking. There is nothing serious wrong here, but I consider this as a drawback of global bundled mode because in my approach, users can use external files as before so no hacking is needed.
  Richard considers this as an advantage because users are given a way to modify files even without lyx. Bo considers this as a bad behavior just like unzip an OOo file, hack an embedded file and zip back.
  It is essential to understand that this proposal distinguishes between bundling and wrapping the bundle. A bundle is just a LyX file with an associate directory that contains the bundled files. Wrapping the bundle is zipping it, or tarring it, or base64 encoding it, or whatever. If you're not wrapped, then you have access to the files almost as normal. If you are, then you don't. So if you are wrapped, then you have no more access than under Bo's proposal, though of course you could unzip/untar/unwhatever the file, do as you wish with it, and then rezip/reetc it. But, if you wanted, you could do that with Bo's too. It'd take five minutes to write a script to do it---actually, I'd hope someone would---but, as someone said, you can also open a LyX file in a text editor and do as you wish. That's the nature of the game.
  What I meant was that in your un-wrapped mode, you are allowing manipulation of bundled files directly
• When a file is saved in compressed zip format, filename.lyxdir is saved in the compressed file. Open such a file will create filename.lyxdir under the temporary directory. What this means is another confusing behavior: when compression is enabled, insert a file will now not insert to $DOC_DIR/filename.lyxdir, but to $TEMP_DIR/filename.lyxdir. (Richard: No, it is not., Bo: Then where should the inserted file go? There is no $DOC_DIR/filename.lyxdir.) This can be confusing when %DOC_DIR/filename.lyxdir exists.
  Just as now, the file would be uncompressed in a temporary directory. Obviously. So this is not an issue.
• Continue from above. if a user turn off compression, $TEMP_DIR/filename.lyxdir will be moved to $DOC_DIR. This may fail because it is possible that filename.lyx is writable but not the document directory. This is a bad consequence of having a 'document' as a file + a directory. Then, if a user turn on compression again, $DOC_DIR/filename.lyxdir remains. Insert a file will insert to $DOC_DIR/filename.lyxdir. Natural? No. Because the file is supposed to be inserted to $TMP_DIR/filename.lyxdir as described above. I do not know how Richard would like to resolve these, which are all 'implementation details', but this is plain ugly.
  None. The path is relative.
  I have no idea what Richard is talking about.
  That would be the problem.
  How I wish you could spare me from guessing how "your" proposal works.
• While embedding class and layout files is easy using the first approach, it is unclear how to do that in this approach. The problem is that local class and layout files need to be in the same directory as the lyx (or exported latex) file. Something similar to $TEXINPUTS has to be used, which complicates the implementation of this method.
• It is unclear to me how to copy an inset from a 'bundled-mode' buffer to a 'normal-mode' buffer. The inset refers to filename_A.lyxdir/figures/blah, which should be copied out of filename_A's directory, but there is no filename_B.lyxdir to accept it.

5.3  Questions and answers regarding Richard's suggestion

6.  Enrico's script proposal

6.1  Idea

An external script is used to probe .lyx file for referred external files, and pack them to a zip file

6.2  Advantages

1. Not intrusive because it is an external script.

6.3  Disadvantages

1. For more advanced features such as embedding layout and class files, some GUI is needed.
2. The bundled file is not directly recognizable by lyx. The recipent of the document needs to use an external program to open the zip file (some OS can open it directly), then edit it.
3. Modifications to the .lyx file is needed to overcome OS differences (c:\blah). Otherwise, there is no gurantee that the document can be compiled on another machine.
4. It is possible to alias all external files. A map can be kept in the bundle so that the bundle can be restored to their original form. Note that restoration is not supposed to be done by a recipent (who only unbundles), but by the original author. How to make sure that the map can be kept when the recipent edit the file and re-bundle is a problem.

7.  General ideas

7.1  Idea 1 - "Edit external file..." in the context menu.

From a post by Richard:

7.2  Idea 2 - Bundling remote files

It could be possible to bundle remote files, e.g.

and then update it when and as necessary.

See the LyXDevelThread:107xxx (help, which post??)

7.3  Idea 3 - Bundling .git or .svn directories

It would make much sense to bundle also .git or .svn directories with the files. then diff or commiting after getting file back from coauthor is just trivial matter.--pavel

7.4  Idea 4 - Automatic inclusion of PDF outut in bundle

See LyXDevelThread:107285 and previous and latter post.

Basically it could work as follows:

• The user enables 'automatic inclusion of latest build PDF' when bundling.
• Whenever a build is successful, the latest build PDF is included in the bundle, replacing a previously PDF.

Motivations: See the parent of LyXDevelThread:107285 .

• You want to send your latest revision of the documents to several recipients for review, and not all of them have LyX.

Implications: This idea speaks for using .zip rather than base64, as it's easier for a typical recipient to extract the PDF from a .zip. /Christian Only the script approach can make use of this implication because the 'product' of that approach is a zip file, and users will explicitly open it and see this PDF file. In the other two approaches, the bundled file has .lyx suffix so lyx will be used to open them. Once the file is uncompressed by lyx, there is not much difference.

8.  Categories

• Development