poky/documentation/README

documentation
=============

This is the directory that contains the Yocto Project documentation.  The Yocto
Project source repositories at https://git.yoctoproject.org/cgit.cgi have two
instances of the "documentation" directory.  You should understand each of
these instances.

  poky/documentation - The directory within the poky Git repository containing
                       the set of Yocto Project manuals.  When you clone the
                       poky Git repository, the documentation directory
                       contains the manuals.  The state of the manuals in this
                       directory is guaranteed to reflect the latest Yocto
                       Project release.  The manuals at the tip of this
                       directory will also likely contain most manual
                       development changes.

  yocto-docs/documentation - The Git repository for the Yocto Project manuals.
                             This repository is where manual development
                             occurs.  If you plan on contributing back to the
                             Yocto Project documentation, you should set up
                             a local Git repository based on this upstream
                             repository as follows:

                               git clone git://git.yoctoproject.org/yocto-docs

                             Changes and patches are first pushed to the
                             yocto-docs Git repository.  Later, they make it
                             into the poky Git repository found at
                             git://git.yoctoproject.org/poky.

Manual Organization
===================

Here the folders corresponding to individual manuals:

* brief-yoctoprojectqs - Yocto Project Quick Start
* overview-manual      - Yocto Project Overview and Concepts Manual
* contributor-guide    - Yocto Project and OpenEmbedded Contributor Guide
* ref-manual           - Yocto Project Reference Manual
* bsp-guide            - Yocto Project Board Support Package (BSP) Developer's Guide
* dev-manual           - Yocto Project Development Tasks Manual
* kernel-dev           - Yocto Project Linux Kernel Development Manual
* profile-manual       - Yocto Project Profiling and Tracing Manual
* sdk-manual           - Yocto Project Software Development Kit (SDK) Developer's Guide.
* toaster-manual       - Toaster User Manual
* test-manual          - Yocto Project Test Environment Manual
* migration-guides     - Yocto Project Release and Migration Notes

Each folder is self-contained regarding content and figures.

If you want to find HTML versions of the Yocto Project manuals on the web,
the current versions reside at https://docs.yoctoproject.org.

poky.yaml
=========

This file defines variables used for documentation production.  The variables
are used to define release pathnames, URLs for the published manuals, etc.

standards.md
============

This file specifies some rules to follow when contributing to the documentation.

template/
=========

Contains a template.svg, to make it easier to create consistent
SVG diagrams.

Sphinx
======

The Yocto Project documentation was migrated from the original DocBook
format to Sphinx based documentation for the Yocto Project 3.2
release. This section will provide additional information related to
the Sphinx migration, and guidelines for developers willing to
contribute to the Yocto Project documentation.

   Sphinx is a tool that makes it easy to create intelligent and
   beautiful documentation, written by Georg Brandl and licensed under
   the BSD license. It was originally created for the Python
   documentation.

Extensive documentation is available on the Sphinx website:
https://www.sphinx-doc.org/en/master/. Sphinx is designed to be
extensible thanks to the ability to write our own custom extensions,
as Python modules, which will be executed during the generation of the
documentation.

Yocto Project documentation website
===================================

The website hosting the Yocto Project documentation, can be found
at: https://docs.yoctoproject.org/.

The entire Yocto Project documentation, as well as the BitBake manual,
is published on this website, including all previously released
versions. A version switcher was added, as a drop-down menu on the top
of the page to switch back and forth between the various versions of
the current active Yocto Project releases.

Transition pages have been added (as rst files) to show links to old
versions of the Yocto Project documentation with links to each manual
generated with DocBook.

How to build the Yocto Project documentation
============================================

To build the documentation, you need Sphinx and a few other packages,
which depend on your host GNU/Linux distribution. Such packages are listed on
https://docs.yoctoproject.org/dev/ref-manual/system-requirements.html#required-packages-for-the-build-host

To build the documentation locally, run:

 $ cd documentation
 $ make html

The resulting HTML index page will be _build/html/index.html, and you
can browse your own copy of the locally generated documentation with
your browser.

Alternatively, you can use Pipenv to automatically install all required
dependencies in a virtual environment:

 $ cd documentation
 $ pipenv install
 $ pipenv run make html

Style checking the Yocto Project documentation
==============================================

The project is starting to use Vale (https://vale.sh/)
to validate the text style.

To install Vale:

 $ pip install vale

To run Vale:

 $ make stylecheck

Style checking the whole documentation might take some time and generate a
lot of warnings/errors, thus one can run Vale on a subset of files or
directories:

  $ make stylecheck VALEDOCS=<file>
  $ make stylecheck VALEDOCS="<file1> <file2>"
  $ make stylecheck VALEDOCS=<dir>

Lint checking the Yocto Project documentation
=============================================

To fix errors which are not reported by Sphinx itself,
the project uses sphinx-lint (https://github.com/sphinx-contrib/sphinx-lint).

To install sphinx-lint:

 $ pip install sphinx-lint

To run sphinx-lint:

 $ make sphinx-lint

Lint checking the whole documentation might take some time and generate a
lot of warnings/errors, thus one can run sphinx-lint on a subset of files
or directories:

  $ make sphinx-lint SPHINXLINTDOCS=<file>
  $ make sphinx-lint SPHINXLINTDOCS="<file1> <file2>"
  $ make sphinx-lint SPHINXLINTDOCS=<dir>

Sphinx theme and CSS customization
==================================

The Yocto Project documentation is currently based on the "Read the
Docs" Sphinx theme, with a few changes to make sure the look and feel
of the project documentation is preserved.

Most of the theme changes can be done using the file
'sphinx-static/theme_overrides.css'. Most CSS changes in this file
were inherited from the DocBook CSS stylesheets.

Sphinx design guidelines and principles
=======================================

The initial Docbook to Sphinx migration was done with an automated
tool called Pandoc (https://pandoc.org/). The tool produced some clean
output markdown text files. After the initial automated conversion
additional changes were done to fix up headings, images and links. In
addition Sphinx has built in mechanisms (directives) which were used
to replace similar functions implemented in Docbook such as glossary,
variables substitutions, notes and references.

Headings
========

The layout of the Yocto Project manuals is organized as follows

    Book
      Chapter
        Section
          Subsection
            Subsubsection
              Subsubsubsection

Here are the heading styles that we use in the manuals:

    Book                       => overline ===
      Chapter                  => overline ***
        Section                => ====
          Subsection           => ----
            Subsubsection      => ~~~~
              Subsubsubsection => ^^^^

With this proposal, we preserve the same TOCs between Sphinx and Docbook.

Built-in glossary
=================

Sphinx has a glossary directive. From
https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#glossary:

    This directive must contain a reST definition list with terms and
    definitions. It's then possible to refer to each definition through the
    [https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-term
    'term' role].

So anywhere in any of the Yocto Project manuals, :term:`VAR` can be
used to refer to an item from the glossary, and a link is created
automatically. A general index of terms is also generated by Sphinx
automatically.

Global substitutions
====================

The Yocto Project documentation makes heavy use of global
variables. In Docbook these variables are stored in the file
poky.ent. This Docbook feature is not handled automatically with
Pandoc. Sphinx has builtin support for substitutions
(https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#substitutions),
however there are important shortcomings. For example they cannot be
used/nested inside code-block sections.

A Sphinx extension was implemented to support variable substitutions
to mimic the DocBook based documentation behavior. Variable
substitutions are done while reading/parsing the .rst files. The
pattern for variables substitutions is the same as with DocBook,
e.g. `&VAR;`.

The implementation of the extension can be found here in the file
documentation/sphinx/yocto-vars.py, this extension is enabled by
default when building the Yocto Project documentation.  All variables
are set in a file call poky.yaml, which was initially generated from
poky.ent. The file was converted into YAML so that it is easier to
process by the custom Sphinx extension (which is a Python module).

For example, the following .rst content will produce the 'expected'
content:

  .. code-block::
      $ mkdir poky-&DISTRO;
      or
      $ git clone &YOCTO_GIT_URL;/git/poky -b &DISTRO_NAME_NO_CAP;

Variables can be nested, like it was the case for DocBook:

  YOCTO_HOME_URL : "https://www.yoctoproject.org"
  YOCTO_DOCS_URL : "&YOCTO_HOME_URL;/docs"

Note directive
==============

Sphinx has a builtin 'note' directive that produces clean Note section
in the output file. There are various types of directives such as
"attention", "caution", "danger", "error", "hint", "important", "tip",
"warning", "admonition" that are supported, and additional directives
can be added as Sphinx extension if needed.

Figures
=======

The Yocto Project documentation has many figures/images. Sphinx has a
'figure' directive which is straightforward to use. To include a
figure in the body of the documentation:

  .. image:: figures/YP-flow-diagram.png

Links and References
====================

The following types of links can be used: links to other locations in
the same document, to locations in other documents and to external
websites.

More information can be found here:
https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html.

For external links, we use this syntax:
`link text <link URL>`__

instead of:
`link text <link URL>`_

Both syntaxes work, but the latter also creates a "link text" reference
target which could conflict with other references with the same name.
So, only use this variant when you wish to make multiple references
to this link, reusing only the target name.

See https://stackoverflow.com/questions/27420317/restructured-text-rst-http-links-underscore-vs-use

Anchor (<#link>) links are forbidden as they are not checked by Sphinx during
the build and may be broken without knowing about it.

References
==========

The following extension is enabled by default:
sphinx.ext.autosectionlabel
(https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html).

This extension allows you to refer sections by their titles. Note that
autosectionlabel_prefix_document is enabled by default, so that we can
insert references from any document.

For example, to insert an HTML link to a section from
documentation/manual/intro.rst, use:

  Please check this :ref:`manual/intro:Cross-References to Locations in the Same Document`

Alternatively a custom text can be used instead of using the section
text:

  Please check this :ref:`section <manual/intro:Cross-References to Locations in the Same Document>`

TIP: The following command can be used to dump all the references that
     are defined in the project documentation:

       python -msphinx.ext.intersphinx <path to build folder>/html/objects.inv

This dump contains all links and for each link it shows the default
"Link Text" that Sphinx would use. If the default link text is not
appropriate, a custom link text can be used in the ':ref:' directive.

Extlinks
========

The sphinx.ext.extlinks extension is enabled by default
(https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html#use-the-external-links-extension),
and it is configured with the 'extlinks' definitions in
the 'documentation/conf.py' file:

  'yocto_home': ('https://yoctoproject.org%s', None),
  'yocto_wiki': ('https://wiki.yoctoproject.org%s', None),
  'yocto_dl': ('https://downloads.yoctoproject.org%s', None),
  'yocto_lists': ('https://lists.yoctoproject.org%s', None),
  'yocto_bugs': ('https://bugzilla.yoctoproject.org%s', None),
  'yocto_ab': ('https://autobuilder.yoctoproject.org%s', None),
  'yocto_docs': ('https://docs.yoctoproject.org%s', None),
  'yocto_git': ('https://git.yoctoproject.org%s', None),
  'oe_home': ('https://www.openembedded.org%s', None),
  'oe_lists': ('https://lists.openembedded.org%s', None),
  'oe_git': ('https://git.openembedded.org%s', None),
  'oe_wiki': ('https://www.openembedded.org/wiki%s', None),
  'oe_layerindex': ('https://layers.openembedded.org%s', None),
  'oe_layer': ('https://layers.openembedded.org/layerindex/branch/master/layer%s', None),

It creates convenient shortcuts which can be used throughout the
documentation rst files, as:

  Please check this :yocto_wiki:`wiki page </Weekly_Status>`

Intersphinx links
=================

The sphinx.ext.intersphinx extension is enabled by default
(https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html),
so that we can cross reference content from other Sphinx based
documentation projects, such as the BitBake manual.

References to the BitBake manual can directly be done:
 - With a specific description instead of the section name:
  :ref:`Azure Storage fetcher (az://) <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`
 - With the section name:
  :ref:`bitbake-user-manual/bitbake-user-manual-intro:usage and syntax` option

If you want to refer to an entire document (or chapter) in the BitBake manual,
you have to use the ":doc:" macro with the "bitbake:" prefix:
 - :doc:`BitBake User Manual <bitbake:index>`
 - :doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter

Note that a reference to a variable (:term:`VARIABLE`) automatically points to
the BitBake manual if the variable is not described in the Reference Manual's Variable Glossary.
However, if you need to bypass this, you can explicitely refer to a description in the
BitBake manual as follows:

  :term:`bitbake:BB_NUMBER_PARSE_THREADS`

This would be the same if we had identical document filenames in
both the Yocto Project and BitBake manuals:

  :ref:`bitbake:directory/file:section title`

Submitting documentation changes
================================

Please see the top level README file in this repository for details of where
to send patches.