@c -*- coding: utf-8; mode: texinfo; -*-
@node LSR work
@chapter LSR work

@menu
* Introduction to LSR::
* Adding and editing snippets::
* Approving snippets::
* The makelsr.pl script::
* LSR to Git::
* Renaming a snippet::
* Updating the LSR to a new version::
@end menu


@node Introduction to LSR
@section Introduction to LSR

The @uref{https://lsr.di.unimi.it/, LilyPond Snippet Repository
(LSR)} is a collection of LilyPond examples.  A subset of these
examples are automatically imported into the documentation, making
it easy for users to contribute to the documentation without
learning Git and Texinfo.


@need 1000
@node Adding and editing snippets
@section Adding and editing snippets

@subheading General guidelines

When you create (or find!) a nice snippet, and if it is supported
by the LilyPond version running on the LSR, please add it to the
LSR.  Go to @uref{https://lsr.di.unimi.it/, LSR} and log in -- if
you haven't already, create an account.  Follow the instructions
on the website.  These instructions also explain how to modify
existing snippets.

If you think a snippet is particularly informative and should be
included in the documentation, tag it with @q{docs} and one or
more other categories, or ask on the development list for somebody
who has editing permissions to do it.

Please make sure that the LilyPond code follows our formatting
guidelines, @pxref{LilyPond formatting}.

If a new snippet created for documentation purposes compiles with
the LilyPond version currently on LSR, it should be added to the
LSR, and a reference to the snippet should be added to the
documentation.  Please ask a documentation editor to add a
reference to it in an appropriate place in the docs.  (Note -- it
should appear in the @q{snippets} document automatically, once it
has been imported into git and built.  @xref{LSR to Git}.)

If a new snippet uses new features that are not available in the
current LSR version of LilyPond, it should be added to directory
@file{Documentation/@/snippets/@/new/}, and a reference should be
added to the manual.

Snippets created or updated in
@file{Documentation/@/snippets/@/new/} must be adjusted and copied
to directory @file{Documentation/@/snippets/}.  This should be done
by invoking the @command{makelsr.pl} script -- @emph{after} you
have compiled LilyPond.  Assuming that your LilyPond build is in
the top-level subdirectory @file{build/}, a proper invocation is

@example
cd /your/lilypond/git/top/dir
scripts/auxiliar/makelsr.pl --new
@end example

@xref{The makelsr.pl script}, for more details.

Be sure that @samp{make doc} runs successfully before submitting a
patch, to prevent breaking compilation (@pxref{Generating
documentation}).


@subheading Formatting snippets in @file{Documentation/snippets/new/}

When adding a file to this directory, please start the file with
the following template @dots{}

@example
\version "2.xx.yy"

\header @{
  % Use existing LSR tags other than 'docs'; the names of the
  % `*.snippet-list` files in `Documentation/snippets/` give the
  % tags currently used.
  lsrtags = "rhythms, expressive-marks"

  % The documentation string must use Texinfo syntax.  In
  % addition, `\` and `"` must be written as `\\` and `\"`,
  % respectively.
  texidoc = "
This snippet demonstrates @@code@{\\foo@} ...
"

  % The snippet title string must be formatted similar to
  % `texidoc`.
  doctitle = "Snippet title"
@}

<LilyPond code starts here>
@end example

@noindent
@dots{} and name the file @file{snippet-title.ly}.

It is important that the version number you use at the top of the
example is the minimum LilyPond version that the file compiles
with: for example, if the LSR is currently at 2.22.2, your example
requires 2.23.4, and the current development version of LilyPond
is @version{}, use @code{\version "2.23.4"}.

Particular attention is also necessary for the @code{lsrtags} and
@code{doctitle} fields: the tags must match tags used in the
documentation, and the @code{doctitle} must match the filename
(@command{makelsr.pl} shows a helpful error message if it
doesn't).

The order of @code{\version}, @code{\header}, and the LilyPond
code must be as shown above, otherwise @command{makelsr.pl} aborts
with an error.  The same holds for the order of the
@code{lsrtags}, @code{texidoc}, and @code{doctitle} fields within
@code{\header}.


@node Approving snippets
@section Approving snippets

The main task of LSR editors is approving snippets.  To find a
list of unapproved snippets, log into
@uref{https://lsr.di.unimi.it/, LSR} and select @qq{No} from the
drop-down menu to the right of the word @qq{Approved} at the
bottom of the interface, then click @qq{Enable filter}.

Here is a checklist of the necessary tasks.

@enumerate

@item
Does the snippet make sense and does it what the author claims
that it does?  If you think the snippet is suited to be included
into the LilyPond documentation, add the @q{docs} tag and at least
one other tag.

@item
If the snippet is tagged with @q{docs}, check whether it matches
our formatting guidelines, @pxref{LilyPond formatting}.

Also, snippets tagged with @q{docs} should not be explaining (or
replicating) existing material in the documentation.  They should
not refer to the documentation; the documentation should rather
refer to them.

@item
If the snippet uses Scheme code, check that everything looks good
and there are no security risks.

@warning{Somebody could add code like @code{#'(system "rm -rf /")}
to a snippet, which would cause catastrophic results if executed!
Take this step @strong{VERY SERIOUSLY}.}

@item
If all is well, check the box labeled @qq{approved} and save the
snippet.

@end enumerate


@node The makelsr.pl script
@section The @command{makelsr.pl} script

As you might have guessed already, @command{makelsr.pl} is a
@uref{https://perl.org, Perl} script.  Obviously, you need Perl to
execute it, which you should now install in case it isn't already
available on your system.

There is a dependency on the @uref{https://pandoc.org, Pandoc}
program, which the script uses to convert LSR's snippet
documentation strings (which are formatted in HTML) to Texinfo.
This must be installed, too.

Furthermore, @command{makelsr.pl} needs a few additional modules
that are not Perl core modules (tested with Perl version 5.36):

@itemize
@item File::Which
@item IPC::Run3
@item MySQL::Dump::Parser::XS
@item Pandoc
@item Parallel::ForkManager
@end itemize

@noindent
Either install missing modules with your package manager (if
available) or use the @command{cpanm} command.@footnote{Most Perl
distributions have this command included; if not, try to install a
package named @q{cpanminus} or having @q{cpanminus} in its name.}

A typical call might me

@example
cpanm --sudo Parallel::ForkManager
@end example

@noindent
to download, compile, and install module
@q{Parallel::@/ForkManager}.@footnote{Note that the program
@command{cpanm} might be called differently; it sometimes has the
Perl version appended to its name, for example
@command{cpanm-5.34}.

The @option{--sudo} option makes the modules install into a system
directory, for example @file{/usr/@/lib/@/perl5/@/site_perl/@/...}
-- you need the superuser password for this.  If you don't want to
do that for whatever reason, just omit @option{--sudo} and follow
the instructions shown in @command{cpanm}'s error message to
install Perl modules locally (i.e., without @command{sudo}
rights).

As of this writing (August 2022) there is a small buglet in a test
of the @q{Pandoc} module that makes it necessary to add option
@option{--notest} for installing this module in case you have to
use @command{cpanm}.}

Finally, it needs to find the @command{convert-ly} script from the
current LilyPond development build.

By default, executing @command{makelsr.pl} performs the following
actions.

@itemize
@item
Download a current MySQL dump of the LSR database (the dump is
regenerated once a day).

@item
Delete all snippet and snippet list files in directory
@file{Documentation/@/snippets/} (but not in
@file{Documentation/@/snippets/@/new/}).

@item
Extract all snippets from the LSR database that have the @q{docs}
tag set, convert their documentation parts from HTML to Texinfo
with the @command{pandoc} program, run the script
@command{convert-ly} to update their LilyPond code parts to
current syntax, and store them in
@file{Documentation/@/snippets/}.

@item
Create snippet list files named @file{winds@/.snippet-@/list} or
@file{connecting-@/notes@/.snippet-@/list} that list the snippets
grouped by tags assigned in the database.  These files are used to
structure LilyPond's @q{snippets} documentation.

@item
Convert all snippet files in
@file{Documentation/@/snippets/@/new/} with @command{convert-ly}
and output them to @file{Documentation/@/snippets/}, possibly
overwriting existing files.
@end itemize

This flow of actions can be adjusted; say
@samp{scripts/auxiliar/makelsr.pl --help} to get a detailed
description of the provided command line options and used
environment variables.


@need 1000
@node LSR to Git
@section LSR to Git

@subheading Introduction

Snippets used in the documentation are in
@file{$LILYPOND_GIT/@/Documentation/@/snippets/}.  This directory
contains a set of all snippets in the LSR that are tagged with
@q{docs}.  An import is done with the @code{makelsr.pl} script,
which downloads a complete database dump of the LSR to update this
directory.

Snippets that are too new to be run on the LSR (which uses a
stable LilyPond version) are put into
@file{$LILYPOND_GIT/@/Documentation/@/snippets/@/new/}.  Once the
LSR gets upgraded to a LilyPond version that can actually compile
them, they are transferred to the LSR and deleted from
@file{snippets/@/new/}.

@q{Git} is the shorthand name for LilyPond's Git repository, which
contains all the development code.  For further information on
setting this up, @pxref{Working with source code}.  An alternative
to setting up a Git repository for people wanting to do LSR work
is to get the source code from
@uref{https://lilypond.org/website/development.html}.  However, we
don't recommend this since it doesn't allow easy submission of
patches as merge requests.


@subheading Importing the LSR to Git

@enumerate

@item
Make sure that the @command{convert-ly} script is a bleeding edge
version -- the latest development release, or even better, freshly
compiled from Git master, with the environment variable
@code{LILYPOND_BUILD_DIR} correctly set up (@pxref{Environment
variables}) in case your build directory isn't
@file{$LILYPOND_GIT/@/build/}.

@item
Check the other prerequisites necessary for executing the
@command{makelsr.pl} script (@pxref{The makelsr.pl script}).

@item
If you are using a git repository, create and check out a branch,
for example

@example
git checkout -b lsr-import
@end example

@item
From the top source directory, execute

@example
scripts/auxiliar/makelsr.pl
@end example

Say @samp{scripts/auxiliar/makelsr.pl --help} to find out how to
modify this call; for example, command line option @option{--dump
@var{file}} makes the script use a locally stored dump file.

@item
Carefully check the output of the script for warnings and errors,
then carefully check the file differences in the git repository.
@samp{git diff} is your friend.

@item
Rebuild the documentation.  If some snippets from
@file{Documentation/@/snippets/} cause the documentation
compilation to fail, try the following steps to fix it.

@itemize

@item
Look up the snippet filename @file{@var{foo}.ly} in the error
output or log file, then fix the file
@file{Documentation/@/snippets/@/@var{foo}.ly} to make the
documentation build successfully.

@item
Determine where it comes from by looking at its first two lines,
e.g., run

@example
head -2 Documentation/snippets/@var{foo}.ly
@end example

@item
If the snippet comes from the LSR, also apply the fix to the
snippet in the LSR and send a notification email to an LSR editor
with CC to the development list, @pxref{Adding and editing
snippets}.

Note that the failure may sometimes not be caused by the snippet
in the LSR but by LilyPond syntax changes that
@command{convert-ly} can't handle automatically.  Such files
@emph{must} be added to the @file{new/} directory.

@item
If the snippet comes from @file{Documentation/@/snippets/@/new/},
apply the fix in
@file{Documentation/@/snippets/@/new/@/@var{foo}.ly} and run
@command{makelsr.pl} as follows:

@example
scripts/auxiliar/makelsr.pl --new
@end example

Then, inspect @file{Documentation/@/snippets/@/@var{foo}.ly} to
check that the fix has been well propagated.

@item
If the build failure was caused by a translation string, you may
have to fix some
@file{Documentation/@/@var{lang}/@/texidocs/@/@var{foo}.texidoc}
files instead.

@end itemize

@item
When you are done, commit your changes to your Git branch and
create a merge request (@pxref{Lifecycle of a merge request}).

@end enumerate


@node Renaming a snippet
@section Renaming a snippet

Due to the potential duality of snippets (i.e., they may exist
both in the LSR database and in
@file{Documentation/@/snippets/@/new/}), this process is a bit
more involved than we might like.

@enumerate
@item
Send an email to an LSR editor, requesting the renaming.

@item
The LSR editor does the renaming (or debates the topic with you),
then warns the LSR-to-git person (wanted: better title) about the
renaming.

@item
LSR-to-git person does his normal job, but then also renames any
copies of the snippets in @file{Documentation/@/snippets/@/new/},
and any instances of the snippet name in the documentation.

@code{git grep} is highly recommended for this task.

@end enumerate


@node Updating the LSR to a new version
@section Updating the LSR to a new version

To update the LSR, perform the following steps:

@enumerate

@item
Start by emailing the LSR maintainer, Sebastiano, and liaising
with him to ensure that updating the snippets is synchronised with
updating the binary running the LSR.

@item
Download the latest snippet tarball from
@uref{https://lsr.di.unimi.it/download/} and extract it.
The relevant files can be found in the @file{all} subdirectory.
Make sure your shell is using an English language version, for
example @code{LANG=en_US}, then run @command{convert-ly} on all
the files.  Use the command-line option @code{--to=version} to
ensure the snippets are updated to the correct stable version.

@item
Make sure that you are using @command{convert-ly} from the latest
available release to gain best advantage from the latest
@code{converting-rules-updates}.

For example:

@itemize

@item
LSR-version: 2.12.2
@item
intended LSR-update to 2.14.2
@item
latest release 2.15.30

@end itemize

Use convert-ly from 2.15.30 and the following terminal command
for all files:

@example
convert-ly -e -t2.14.2 *.ly
@end example

@item
There might be no conversion rule for some old commands. To make
an initial check for possible problems you can run the
script at the end of this list on a copy of the @file{all} subdirectory.

@item
Copy relevant snippets (i.e., snippets whose version is equal to
or less than the new version of LilyPond running on the LSR) from
@file{Documentation/snippets/new/} into the set of files to be
used to make the tarball.  Make sure
you only choose snippets which are already present in the LSR,
since the LSR software isn't able to create new snippets this way.
If you don't have a Git repository for LilyPond, you'll find these
snippets in the source-tarball on
@uref{https://lilypond.org/website/development.html}.
Don't rename any files at this stage.

@item
Verify that all files compile with the new version of LilyPond,
ideally without any warnings or errors.  To ease the process,
you may use the shell script that appears after this list.

Due to the workload involved, we @emph{do not} require that you
verify that all snippets produce the expected output.  If you
happen to notice any such snippets and can fix them, great; but as
long as all snippets compile, don't delay this step due to some
weird output.  If a snippet is not compiling, update it manually.
If it's not possible, delete it for now.

@item
Remove all headers and version-statements from the files.  Phil
Holmes has a @code{python} script that will do this and which
needs testing.  Please ask him for a copy if you wish to do this.

@item
Create a tarball and send it back to Sebastiano. Don't forget to
tell him about any deletions.

@item
Use the LSR web interface to change any descriptions you want to.
Changing the titles of snippets is a bit fraught, since this also
changes the filenames.  Only do this as a last resort.

@item
Use the LSR web interface to add the other snippets from
@file{Documentation/snippets/new/} which compile with the new
LilyPond version of the LSR.  Ensure that they are correctly
tagged, including the tag @code{docs} and that they are approved.

@item
When LSR has been updated, wait a day for the tarball to update,
then download another snippet tarball. Verify that
the relevant snippets from @file{Documentation/snippets/new/} are
now included, then delete those snippets from
@file{Documentation/snippets/new/}.

@item
Commit all the changes.  @emph{Don't forget to add new files to
the git repository with @code{git add}}.  Run @code{make},
@code{make doc} and
@code{make test} to ensure the changes don't break the build. Any
snippets that have had their file name changed or have been
deleted could break the build, and these will need correcting
step by step.

@end enumerate


Below is a shell script to run LilyPond on all @file{.ly} files in a directory.
If the script is run with a -s parameter, it runs silently except for reporting
failed files.  If run with -c it also runs @code{convert-ly} prior to running
LilyPond.

@smallexample
#!/bin/bash

while getopts sc opt; do
    case $opt in
        s)
            silent=true
            ;;
        c)
            convert=true
            ;;
    esac
done
param=$@
if [ $silent ]; then
    param=$@{param:3@}
fi
if [ $convert ]; then
    param=$@{param:3@}
fi
filter=$@{param:-"*.ly"@}

for LILYFILE in $filter
do
    STEM=$(basename "$LILYFILE" .ly)
    if [ $convert ]; then
        if [ $silent ]; then
            $LILYPOND_BUILD_DIR/out/bin/convert-ly -e "$LILYFILE" >& "$STEM".con.txt
        else
            $LILYPOND_BUILD_DIR/out/bin/convert-ly -e "$LILYFILE"
        fi
    fi
    if [ ! $silent ]; then
        echo "running $LILYFILE..."
    fi
    $LILYPOND_BUILD_DIR/out/bin/lilypond --format=png "$LILYFILE" >& "$STEM".txt
    RetVal=$?
    if [ $RetVal -gt 0 ]; then
       echo "$LILYFILE failed"
    fi
done
@end smallexample

Output from LilyPond is in @file{filename.txt} and convert-ly in
@file{filename.con.txt}.
