technical-details.xml 66 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358
  1. <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
  2. "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
  3. [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
  4. <chapter id='technical-details'>
  5. <title>Technical Details</title>
  6. <para>
  7. This chapter provides technical details for various parts of the Yocto Project.
  8. Currently, topics include Yocto Project components,
  9. shared state (sstate) cache, x32, and Licenses.
  10. </para>
  11. <section id='usingpoky-components'>
  12. <title>Yocto Project Components</title>
  13. <para>
  14. The BitBake task executor together with various types of configuration files form the
  15. OpenEmbedded Core.
  16. This section overviews these by describing what they are used for
  17. and how they interact.
  18. </para>
  19. <para>
  20. BitBake handles the parsing and execution of the data files.
  21. The data itself is of various types:
  22. <itemizedlist>
  23. <listitem><para><emphasis>Recipes:</emphasis> Provides details about particular
  24. pieces of software.</para></listitem>
  25. <listitem><para><emphasis>Class Data:</emphasis> Abstracts common build
  26. information (e.g. how to build a Linux kernel).</para></listitem>
  27. <listitem><para><emphasis>Configuration Data:</emphasis> Defines machine-specific settings,
  28. policy decisions, and so forth.
  29. Configuration data acts as the glue to bind everything together.</para></listitem>
  30. </itemizedlist>
  31. For more information on data, see the
  32. "<ulink url='&YOCTO_DOCS_DEV_URL;#yocto-project-terms'>Yocto Project Terms</ulink>"
  33. section in the Yocto Project Development Manual.
  34. </para>
  35. <para>
  36. BitBake knows how to combine multiple data sources together and refers to each data source
  37. as a layer.
  38. For information on layers, see the
  39. "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and
  40. Creating Layers</ulink>" section of the Yocto Project Development Manual.
  41. </para>
  42. <para>
  43. Following are some brief details on these core components.
  44. For more detailed information on these components, see the
  45. "<link linkend='ref-structure'>Source Directory Structure</link>" chapter.
  46. </para>
  47. <section id='usingpoky-components-bitbake'>
  48. <title>BitBake</title>
  49. <para>
  50. BitBake is the tool at the heart of the OpenEmbedded build system
  51. and is responsible for parsing the
  52. <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink>,
  53. generating a list of tasks from it, and then executing those tasks.
  54. To see a list of the options BitBake supports, use either of
  55. the following commands:
  56. <literallayout class='monospaced'>
  57. $ bitbake -h
  58. $ bitbake --help
  59. </literallayout>
  60. </para>
  61. <para>
  62. The most common usage for BitBake is <filename>bitbake &lt;packagename&gt;</filename>, where
  63. <filename>packagename</filename> is the name of the package you want to build
  64. (referred to as the "target" in this manual).
  65. The target often equates to the first part of a <filename>.bb</filename> filename.
  66. So, to process the <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you
  67. might type the following:
  68. <literallayout class='monospaced'>
  69. $ bitbake matchbox-desktop
  70. </literallayout>
  71. Several different versions of <filename>matchbox-desktop</filename> might exist.
  72. BitBake chooses the one selected by the distribution configuration.
  73. You can get more details about how BitBake chooses between different
  74. target versions and providers in the
  75. "<link linkend='ref-bitbake-providers'>Preferences and Providers</link>" section.
  76. </para>
  77. <para>
  78. BitBake also tries to execute any dependent tasks first.
  79. So for example, before building <filename>matchbox-desktop</filename>, BitBake
  80. would build a cross compiler and <filename>eglibc</filename> if they had not already
  81. been built.
  82. <note>This release of the Yocto Project does not support the <filename>glibc</filename>
  83. GNU version of the Unix standard C library. By default, the OpenEmbedded build system
  84. builds with <filename>eglibc</filename>.</note>
  85. </para>
  86. <para>
  87. A useful BitBake option to consider is the <filename>-k</filename> or
  88. <filename>--continue</filename> option.
  89. This option instructs BitBake to try and continue processing the job as much
  90. as possible even after encountering an error.
  91. When an error occurs, the target that
  92. failed and those that depend on it cannot be remade.
  93. However, when you use this option other dependencies can still be processed.
  94. </para>
  95. </section>
  96. <section id='usingpoky-components-metadata'>
  97. <title>Metadata (Recipes)</title>
  98. <para>
  99. The <filename>.bb</filename> files are usually referred to as "recipes."
  100. In general, a recipe contains information about a single piece of software.
  101. This information includes the location from which to download the
  102. unaltered source, any source patches to be applied to that source
  103. (if needed), which special configuration options to apply,
  104. how to compile the source files, and how to package the compiled output.
  105. </para>
  106. <para>
  107. The term "package" is sometimes used to refer to recipes. However,
  108. since the word "package" is used for the packaged output from the OpenEmbedded
  109. build system (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
  110. this document avoids using the term "package" when referring to recipes.
  111. </para>
  112. </section>
  113. <section id='usingpoky-components-classes'>
  114. <title>Classes</title>
  115. <para>
  116. Class files (<filename>.bbclass</filename>) contain information that
  117. is useful to share between
  118. <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> files.
  119. An example is the Autotools class, which contains
  120. common settings for any application that Autotools uses.
  121. The "<link linkend='ref-classes'>Classes</link>" chapter provides details
  122. about common classes and how to use them.
  123. </para>
  124. </section>
  125. <section id='usingpoky-components-configuration'>
  126. <title>Configuration</title>
  127. <para>
  128. The configuration files (<filename>.conf</filename>) define various configuration variables
  129. that govern the OpenEmbedded build process.
  130. These files fall into several areas that define machine configuration options,
  131. distribution configuration options, compiler tuning options, general common configuration
  132. options, and user configuration options in <filename>local.conf</filename>, which is found
  133. in the
  134. <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
  135. </para>
  136. </section>
  137. </section>
  138. <section id="cross-development-toolchain-generation">
  139. <title>Cross-Development Toolchain Generation</title>
  140. <para>
  141. The Yocto Project does most of the work for you when it comes to
  142. creating
  143. <ulink url='&YOCTO_DOCS_DEV_URL;#cross-development-toolchain'>cross-development toolchains</ulink>.
  144. This section provides some technical background information on how
  145. cross-development toolchains are created and used.
  146. For more information on toolchains, you can also see the
  147. <ulink url='&YOCTO_DOCS_ADT_URL;'>the Yocto Project Application Developer's Guide</ulink>.
  148. </para>
  149. <para>
  150. In the Yocto Project development environment, cross-development
  151. toolchains are used to build the image and applications that run on the
  152. target hardware.
  153. With just a few commands, the OpenEmbedded build system creates
  154. these necessary toolchains for you.
  155. </para>
  156. <para>
  157. The following figure shows a high-level build environment regarding
  158. toolchain construction and use.
  159. </para>
  160. <para>
  161. <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" />
  162. </para>
  163. <para>
  164. Most of the work occurs on the Build Host.
  165. This is the machine used to build images and generally work within the
  166. the Yocto Project environment.
  167. When you run BitBake to create an image, the OpenEmbedded build system
  168. uses the host <filename>gcc</filename> compiler to bootstrap a
  169. cross-compiler named <filename>gcc-cross</filename>.
  170. The <filename>gcc-cross</filename> compiler is what BitBake uses to
  171. compile source files when creating the target image.
  172. You can think of <filename>gcc-cross</filename> simply as an
  173. automatically generated cross-compiler that is used internally within
  174. BitBake only.
  175. </para>
  176. <para>
  177. The chain of events that occurs when <filename>gcc-cross</filename> is
  178. bootstrapped is as follows:
  179. <literallayout class='monospaced'>
  180. gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> eglibc-initial -> eglibc -> gcc-cross -> gcc-runtime
  181. </literallayout>
  182. <itemizedlist>
  183. <listitem><para><filename>gcc</filename>:
  184. The build host's GNU Compiler Collection (GCC).
  185. </para></listitem>
  186. <listitem><para><filename>binutils-cross</filename>:
  187. The bare minimum binary utilities needed in order to run
  188. the <filename>gcc-cross-initial</filename> phase of the
  189. bootstrap operation.
  190. </para></listitem>
  191. <listitem><para><filename>gcc-cross-initial</filename>:
  192. An early stage of the bootstrap process for creating
  193. the cross-compiler.
  194. This stage builds enough of the <filename>gcc-cross</filename>,
  195. the C library, and other pieces needed to finish building the
  196. final cross-compiler in later stages.
  197. This tool is a "native" package (i.e. it is designed to run on
  198. the build host).
  199. </para></listitem>
  200. <listitem><para><filename>linux-libc-headers</filename>:
  201. Headers needed for the cross-compiler.
  202. </para></listitem>
  203. <listitem><para><filename>eglibc-initial</filename>:
  204. An initial version of the Embedded GLIBC needed to bootstrap
  205. <filename>eglibc</filename>.
  206. </para></listitem>
  207. <listitem><para><filename>gcc-cross</filename>:
  208. The final stage of the bootstrap process for the
  209. cross-compiler.
  210. This stage results in the actual cross-compiler that
  211. BitBake uses when it builds an image for a targeted
  212. device.
  213. <note>
  214. If you are replacing this cross compiler toolchain
  215. with a custom version, you must replace
  216. <filename>gcc-cross</filename>.
  217. </note>
  218. This tool is also a "native" package (i.e. it is
  219. designed to run on the build host).
  220. </para></listitem>
  221. <listitem><para><filename>gcc-runtime</filename>:
  222. Runtime libraries resulting from the toolchain bootstrapping
  223. process.
  224. This tool produces a binary that consists of the
  225. runtime libraries need for the targeted device.
  226. </para></listitem>
  227. </itemizedlist>
  228. </para>
  229. <para>
  230. You can use the OpenEmbedded build system to build an installer for
  231. the relocatable SDK used to develop applications.
  232. When you run the installer, it installs the toolchain, which contains
  233. the development tools (e.g., the
  234. <filename>gcc-cross-canadian</filename>),
  235. <filename>binutils-cross-canadian</filename>, and other
  236. <filename>nativesdk-*</filename> tools you need to cross-compile and
  237. test your software.
  238. The figure shows the commands you use to easily build out this
  239. toolchain.
  240. This cross-development toolchain is built to execute on the
  241. <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>,
  242. which might or might not be the same
  243. machine as the Build Host.
  244. <note>
  245. If your target architecture is supported by the Yocto Project,
  246. you can take advantage of pre-built images that ship with the
  247. Yocto Project and already contain cross-development toolchain
  248. installers.
  249. </note>
  250. </para>
  251. <para>
  252. Here is the bootstrap process for the relocatable toolchain:
  253. <literallayout class='monospaced'>
  254. gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> eglibc-initial -> nativesdk-eglibc -> gcc-crosssdk -> gcc-cross-canadian
  255. </literallayout>
  256. <itemizedlist>
  257. <listitem><para><filename>gcc</filename>:
  258. The build host's GNU Compiler Collection (GCC).
  259. </para></listitem>
  260. <listitem><para><filename>binutils-crosssdk</filename>:
  261. The bare minimum binary utilities needed in order to run
  262. the <filename>gcc-crosssdk-initial</filename> phase of the
  263. bootstrap operation.
  264. </para></listitem>
  265. <listitem><para><filename>gcc-crosssdk-initial</filename>:
  266. An early stage of the bootstrap process for creating
  267. the cross-compiler.
  268. This stage builds enough of the
  269. <filename>gcc-crosssdk</filename> and supporting pieces so that
  270. the final stage of the bootstrap process can produce the
  271. finished cross-compiler.
  272. This tool is a "native" binary that runs on the build host.
  273. </para></listitem>
  274. <listitem><para><filename>linux-libc-headers</filename>:
  275. Headers needed for the cross-compiler.
  276. </para></listitem>
  277. <listitem><para><filename>eglibc-initial</filename>:
  278. An initial version of the Embedded GLIBC needed to bootstrap
  279. <filename>nativesdk-eglibc</filename>.
  280. </para></listitem>
  281. <listitem><para><filename>nativesdk-eglibc</filename>:
  282. The Embedded GLIBC needed to bootstrap the
  283. <filename>gcc-crosssdk</filename>.
  284. </para></listitem>
  285. <listitem><para><filename>gcc-crosssdk</filename>:
  286. The final stage of the bootstrap process for the
  287. relocatable cross-compiler.
  288. The <filename>gcc-crosssdk</filename> is a transitory compiler
  289. and never leaves the build host.
  290. Its purpose is to help in the bootstrap process to create the
  291. eventual relocatable <filename>gcc-cross-canadian</filename>
  292. compiler, which is relocatable.
  293. This tool is also a "native" package (i.e. it is
  294. designed to run on the build host).
  295. </para></listitem>
  296. <listitem><para><filename>gcc-cross-canadian</filename>:
  297. The final relocatable cross-compiler.
  298. When run on the
  299. <link linkend='var-SDKMACHINE'><filename>SDKMACHINE</filename></link>,
  300. this tool
  301. produces executable code that runs on the target device.
  302. </para></listitem>
  303. </itemizedlist>
  304. </para>
  305. </section>
  306. <section id="shared-state-cache">
  307. <title>Shared State Cache</title>
  308. <para>
  309. By design, the OpenEmbedded build system builds everything from scratch unless
  310. BitBake can determine that parts do not need to be rebuilt.
  311. Fundamentally, building from scratch is attractive as it means all parts are
  312. built fresh and there is no possibility of stale data causing problems.
  313. When developers hit problems, they typically default back to building from scratch
  314. so they know the state of things from the start.
  315. </para>
  316. <para>
  317. Building an image from scratch is both an advantage and a disadvantage to the process.
  318. As mentioned in the previous paragraph, building from scratch ensures that
  319. everything is current and starts from a known state.
  320. However, building from scratch also takes much longer as it generally means
  321. rebuilding things that do not necessarily need to be rebuilt.
  322. </para>
  323. <para>
  324. The Yocto Project implements shared state code that supports incremental builds.
  325. The implementation of the shared state code answers the following questions that
  326. were fundamental roadblocks within the OpenEmbedded incremental build support system:
  327. <itemizedlist>
  328. <listitem><para>What pieces of the system have changed and what pieces have
  329. not changed?</para></listitem>
  330. <listitem><para>How are changed pieces of software removed and replaced?</para></listitem>
  331. <listitem><para>How are pre-built components that do not need to be rebuilt from scratch
  332. used when they are available?</para></listitem>
  333. </itemizedlist>
  334. </para>
  335. <para>
  336. For the first question, the build system detects changes in the "inputs" to a given task by
  337. creating a checksum (or signature) of the task's inputs.
  338. If the checksum changes, the system assumes the inputs have changed and the task needs to be
  339. rerun.
  340. For the second question, the shared state (sstate) code tracks which tasks add which output
  341. to the build process.
  342. This means the output from a given task can be removed, upgraded or otherwise manipulated.
  343. The third question is partly addressed by the solution for the second question
  344. assuming the build system can fetch the sstate objects from remote locations and
  345. install them if they are deemed to be valid.
  346. </para>
  347. <note>
  348. The OpenEmbedded build system does not maintain
  349. <link linkend='var-PR'><filename>PR</filename></link> information
  350. as part of the shared state packages.
  351. Consequently, considerations exist that affect maintaining shared
  352. state feeds.
  353. For information on how the OpenEmbedded works with packages and can
  354. track incrementing <filename>PR</filename> information, see the
  355. "<ulink url='&YOCTO_DOCS_DEV_URL;#incrementing-a-package-revision-number'>Incrementing a Package Revision Number</ulink>"
  356. section.
  357. </note>
  358. <para>
  359. The rest of this section goes into detail about the overall incremental build
  360. architecture, the checksums (signatures), shared state, and some tips and tricks.
  361. </para>
  362. <section id='overall-architecture'>
  363. <title>Overall Architecture</title>
  364. <para>
  365. When determining what parts of the system need to be built, BitBake
  366. works on a per-task basis rather than a per-recipe basis.
  367. You might wonder why using a per-task basis is preferred over a per-recipe basis.
  368. To help explain, consider having the IPK packaging backend enabled and then switching to DEB.
  369. In this case, <filename>do_install</filename> and <filename>do_package</filename>
  370. outputs are still valid.
  371. However, with a per-recipe approach, the build would not include the
  372. <filename>.deb</filename> files.
  373. Consequently, you would have to invalidate the whole build and rerun it.
  374. Rerunning everything is not the best solution.
  375. Also, in this case, the core must be "taught" much about specific tasks.
  376. This methodology does not scale well and does not allow users to easily add new tasks
  377. in layers or as external recipes without touching the packaged-staging core.
  378. </para>
  379. </section>
  380. <section id='checksums'>
  381. <title>Checksums (Signatures)</title>
  382. <para>
  383. The shared state code uses a checksum, which is a unique signature of a task's
  384. inputs, to determine if a task needs to be run again.
  385. Because it is a change in a task's inputs that triggers a rerun, the process
  386. needs to detect all the inputs to a given task.
  387. For shell tasks, this turns out to be fairly easy because
  388. the build process generates a "run" shell script for each task and
  389. it is possible to create a checksum that gives you a good idea of when
  390. the task's data changes.
  391. </para>
  392. <para>
  393. To complicate the problem, there are things that should not be included in
  394. the checksum.
  395. First, there is the actual specific build path of a given task -
  396. the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
  397. It does not matter if the working directory changes because it should not
  398. affect the output for target packages.
  399. Also, the build process has the objective of making native or cross packages relocatable.
  400. The checksum therefore needs to exclude <filename>WORKDIR</filename>.
  401. The simplistic approach for excluding the working directory is to set
  402. <filename>WORKDIR</filename> to some fixed value and create the checksum
  403. for the "run" script.
  404. </para>
  405. <para>
  406. Another problem results from the "run" scripts containing functions that
  407. might or might not get called.
  408. The incremental build solution contains code that figures out dependencies
  409. between shell functions.
  410. This code is used to prune the "run" scripts down to the minimum set,
  411. thereby alleviating this problem and making the "run" scripts much more
  412. readable as a bonus.
  413. </para>
  414. <para>
  415. So far we have solutions for shell scripts.
  416. What about Python tasks?
  417. The same approach applies even though these tasks are more difficult.
  418. The process needs to figure out what variables a Python function accesses
  419. and what functions it calls.
  420. Again, the incremental build solution contains code that first figures out
  421. the variable and function dependencies, and then creates a checksum for the data
  422. used as the input to the task.
  423. </para>
  424. <para>
  425. Like the <filename>WORKDIR</filename> case, situations exist where dependencies
  426. should be ignored.
  427. For these cases, you can instruct the build process to ignore a dependency
  428. by using a line like the following:
  429. <literallayout class='monospaced'>
  430. PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
  431. </literallayout>
  432. This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
  433. depend on the value of <filename>MACHINE</filename>, even if it does reference it.
  434. </para>
  435. <para>
  436. Equally, there are cases where we need to add dependencies BitBake is not able to find.
  437. You can accomplish this by using a line like the following:
  438. <literallayout class='monospaced'>
  439. PACKAGE_ARCHS[vardeps] = "MACHINE"
  440. </literallayout>
  441. This example explicitly adds the <filename>MACHINE</filename> variable as a
  442. dependency for <filename>PACKAGE_ARCHS</filename>.
  443. </para>
  444. <para>
  445. Consider a case with in-line Python, for example, where BitBake is not
  446. able to figure out dependencies.
  447. When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
  448. produces output when it discovers something for which it cannot figure out
  449. dependencies.
  450. The Yocto Project team has currently not managed to cover those dependencies
  451. in detail and is aware of the need to fix this situation.
  452. </para>
  453. <para>
  454. Thus far, this section has limited discussion to the direct inputs into a task.
  455. Information based on direct inputs is referred to as the "basehash" in the
  456. code.
  457. However, there is still the question of a task's indirect inputs - the
  458. things that were already built and present in the
  459. <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
  460. The checksum (or signature) for a particular task needs to add the hashes
  461. of all the tasks on which the particular task depends.
  462. Choosing which dependencies to add is a policy decision.
  463. However, the effect is to generate a master checksum that combines the basehash
  464. and the hashes of the task's dependencies.
  465. </para>
  466. <para>
  467. At the code level, there are a variety of ways both the basehash and the
  468. dependent task hashes can be influenced.
  469. Within the BitBake configuration file, we can give BitBake some extra information
  470. to help it construct the basehash.
  471. The following statement effectively results in a list of global variable
  472. dependency excludes - variables never included in any checksum:
  473. <literallayout class='monospaced'>
  474. BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
  475. SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
  476. USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
  477. PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
  478. CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
  479. </literallayout>
  480. The previous example excludes
  481. <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
  482. since that variable is actually constructed as a path within
  483. <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on
  484. the whitelist.
  485. </para>
  486. <para>
  487. The rules for deciding which hashes of dependent tasks to include through
  488. dependency chains are more complex and are generally accomplished with a
  489. Python function.
  490. The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
  491. of this and also illustrates how you can insert your own policy into the system
  492. if so desired.
  493. This file defines the two basic signature generators <filename>OE-Core</filename>
  494. uses: "OEBasic" and "OEBasicHash".
  495. By default, there is a dummy "noop" signature handler enabled in BitBake.
  496. This means that behavior is unchanged from previous versions.
  497. <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
  498. through this setting in the <filename>bitbake.conf</filename> file:
  499. <literallayout class='monospaced'>
  500. BB_SIGNATURE_HANDLER ?= "OEBasicHash"
  501. </literallayout>
  502. The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
  503. "OEBasic" version but adds the task hash to the stamp files.
  504. This results in any
  505. <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink>
  506. change that changes the task hash, automatically
  507. causing the task to be run again.
  508. This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link>
  509. values, and changes to Metadata automatically ripple across the build.
  510. </para>
  511. <para>
  512. It is also worth noting that the end result of these signature generators is to
  513. make some dependency and hash information available to the build.
  514. This information includes:
  515. <literallayout class='monospaced'>
  516. BB_BASEHASH_task-&lt;taskname&gt; - the base hashes for each task in the recipe
  517. BB_BASEHASH_&lt;filename:taskname&gt; - the base hashes for each dependent task
  518. BBHASHDEPS_&lt;filename:taskname&gt; - The task dependencies for each task
  519. BB_TASKHASH - the hash of the currently running task
  520. </literallayout>
  521. </para>
  522. </section>
  523. <section id='shared-state'>
  524. <title>Shared State</title>
  525. <para>
  526. Checksums and dependencies, as discussed in the previous section, solve half the
  527. problem of supporting a shared state.
  528. The other part of the problem is being able to use checksum information during the build
  529. and being able to reuse or rebuild specific components.
  530. </para>
  531. <para>
  532. The shared state class (<filename>sstate.bbclass</filename>)
  533. is a relatively generic implementation of how to "capture" a snapshot of a given task.
  534. The idea is that the build process does not care about the source of a task's output.
  535. Output could be freshly built or it could be downloaded and unpacked from
  536. somewhere - the build process does not need to worry about its origin.
  537. </para>
  538. <para>
  539. There are two types of output, one is just about creating a directory
  540. in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
  541. A good example is the output of either <filename>do_install</filename> or
  542. <filename>do_package</filename>.
  543. The other type of output occurs when a set of data is merged into a shared directory
  544. tree such as the sysroot.
  545. </para>
  546. <para>
  547. The Yocto Project team has tried to keep the details of the implementation hidden in
  548. <filename>sstate.bbclass</filename>.
  549. From a user's perspective, adding shared state wrapping to a task
  550. is as simple as this <filename>do_deploy</filename> example taken from
  551. <filename>deploy.bbclass</filename>:
  552. <literallayout class='monospaced'>
  553. DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
  554. SSTATETASKS += "do_deploy"
  555. do_deploy[sstate-name] = "deploy"
  556. do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
  557. do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
  558. python do_deploy_setscene () {
  559. sstate_setscene(d)
  560. }
  561. addtask do_deploy_setscene
  562. do_deploy[dirs] = "${DEPLOYDIR} ${B}"
  563. </literallayout>
  564. In this example, we add some extra flags to the task, a name field ("deploy"), an
  565. input directory where the task sends data, and the output
  566. directory where the data from the task should eventually be copied.
  567. We also add a <filename>_setscene</filename> variant of the task and add the task
  568. name to the <filename>SSTATETASKS</filename> list.
  569. </para>
  570. <para>
  571. If you have a directory whose contents you need to preserve, you can do this with
  572. a line like the following:
  573. <literallayout class='monospaced'>
  574. do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
  575. </literallayout>
  576. This method, as well as the following example, also works for multiple directories.
  577. <literallayout class='monospaced'>
  578. do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
  579. do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
  580. do_package[sstate-lockfile] = "${PACKAGELOCK}"
  581. </literallayout>
  582. These methods also include the ability to take a lockfile when manipulating
  583. shared state directory structures since some cases are sensitive to file
  584. additions or removals.
  585. </para>
  586. <para>
  587. Behind the scenes, the shared state code works by looking in
  588. <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> and
  589. <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link>
  590. for shared state files.
  591. Here is an example:
  592. <literallayout class='monospaced'>
  593. SSTATE_MIRRORS ?= "\
  594. file://.* http://someserver.tld/share/sstate/PATH \n \
  595. file://.* file:///some/local/dir/sstate/PATH"
  596. </literallayout>
  597. <note>
  598. The shared state directory (<filename>SSTATE_DIR</filename>) is
  599. organized into two-character subdirectories, where the subdirectory
  600. names are based on the first two characters of the hash.
  601. If the shared state directory structure for a mirror has the
  602. same structure as <filename>SSTATE_DIR</filename>, you must
  603. specify "PATH" as part of the URI to enable the build system
  604. to map to the appropriate subdirectory.
  605. </note>
  606. </para>
  607. <para>
  608. The shared state package validity can be detected just by looking at the
  609. filename since the filename contains the task checksum (or signature) as
  610. described earlier in this section.
  611. If a valid shared state package is found, the build process downloads it
  612. and uses it to accelerate the task.
  613. </para>
  614. <para>
  615. The build processes use the <filename>*_setscene</filename> tasks
  616. for the task acceleration phase.
  617. BitBake goes through this phase before the main execution code and tries
  618. to accelerate any tasks for which it can find shared state packages.
  619. If a shared state package for a task is available, the shared state
  620. package is used.
  621. This means the task and any tasks on which it is dependent are not
  622. executed.
  623. </para>
  624. <para>
  625. As a real world example, the aim is when building an IPK-based image,
  626. only the <filename>do_package_write_ipk</filename> tasks would have their
  627. shared state packages fetched and extracted.
  628. Since the sysroot is not used, it would never get extracted.
  629. This is another reason why a task-based approach is preferred over a
  630. recipe-based approach, which would have to install the output from every task.
  631. </para>
  632. </section>
  633. <section id='tips-and-tricks'>
  634. <title>Tips and Tricks</title>
  635. <para>
  636. The code in the build system that supports incremental builds is not
  637. simple code.
  638. This section presents some tips and tricks that help you work around
  639. issues related to shared state code.
  640. </para>
  641. <section id='debugging'>
  642. <title>Debugging</title>
  643. <para>
  644. When things go wrong, debugging needs to be straightforward.
  645. Because of this, the Yocto Project team included strong debugging
  646. tools:
  647. <itemizedlist>
  648. <listitem><para>Whenever a shared state package is written out, so is a
  649. corresponding <filename>.siginfo</filename> file.
  650. This practice results in a pickled Python database of all
  651. the metadata that went into creating the hash for a given shared state
  652. package.</para></listitem>
  653. <listitem><para>If you run BitBake with the <filename>--dump-signatures</filename>
  654. (or <filename>-S</filename>) option, BitBake dumps out
  655. <filename>.siginfo</filename> files in
  656. the stamp directory for every task it would have executed instead of
  657. building the specified target package.</para></listitem>
  658. <listitem><para>There is a <filename>bitbake-diffsigs</filename> command that
  659. can process <filename>.siginfo</filename> files.
  660. If you specify one of these files, BitBake dumps out the dependency
  661. information in the file.
  662. If you specify two files, BitBake compares the two files and dumps out
  663. the differences between the two.
  664. This more easily helps answer the question of "What
  665. changed between X and Y?"</para></listitem>
  666. </itemizedlist>
  667. </para>
  668. </section>
  669. <section id='invalidating-shared-state'>
  670. <title>Invalidating Shared State</title>
  671. <para>
  672. The OpenEmbedded build system uses checksums and shared state
  673. cache to avoid unnecessarily rebuilding tasks.
  674. Collectively, this scheme is known as "shared state code."
  675. </para>
  676. <para>
  677. As with all schemes, this one has some drawbacks.
  678. It is possible that you could make implicit changes to your
  679. code that the checksum calculations do not take into
  680. account (i.e. implicit changes).
  681. These implicit changes affect a task's output but do not trigger
  682. the shared state code into rebuilding a recipe.
  683. Consider an example during which a tool changes its output.
  684. Assume that the output of <filename>rpmdeps</filename> changes.
  685. The result of the change should be that all the
  686. <filename>package</filename> and
  687. <filename>package_write_rpm</filename> shared state cache
  688. items become invalid.
  689. However, because the change to the output is
  690. external to the code and therefore implicit,
  691. the associated shared state cache items do not become
  692. invalidated.
  693. In this case, the build process uses the cached items rather
  694. than running the task again.
  695. Obviously, these types of implicit changes can cause problems.
  696. </para>
  697. <para>
  698. To avoid these problems during the build, you need to
  699. understand the effects of any changes you make.
  700. Realize that changes you make directly to a function
  701. are automatically factored into the checksum calculation.
  702. Thus, these explicit changes invalidate the associated area of
  703. sstate cache.
  704. However, you need to be aware of any implicit changes that
  705. are not obvious changes to the code and could affect the output
  706. of a given task.
  707. </para>
  708. <para>
  709. When you identify an implicit change, you can easily take steps
  710. to invalidate the cache and force the tasks to run.
  711. The steps you can take are as simple as changing a function's
  712. comments in the source code.
  713. For example, to invalidate package shared state files, change
  714. the comment statements of <filename>do_package</filename> or
  715. the comments of one of the functions it calls.
  716. Even though the change is purely cosmetic, it causes the
  717. checksum to be recalculated and forces the OpenEmbedded build
  718. system to run the task again.
  719. </para>
  720. <note>
  721. For an example of a commit that makes a cosmetic change to
  722. invalidate shared state, see this
  723. <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
  724. </note>
  725. </section>
  726. </section>
  727. </section>
  728. <section id='x32'>
  729. <title>x32</title>
  730. <para>
  731. x32 is a processor-specific Application Binary Interface (psABI) for x86_64.
  732. An ABI defines the calling conventions between functions in a processing environment.
  733. The interface determines what registers are used and what the sizes are for various C data types.
  734. </para>
  735. <para>
  736. Some processing environments prefer using 32-bit applications even when running
  737. on Intel 64-bit platforms.
  738. Consider the i386 psABI, which is a very old 32-bit ABI for Intel 64-bit platforms.
  739. The i386 psABI does not provide efficient use and access of the Intel 64-bit processor resources,
  740. leaving the system underutilized.
  741. Now consider the x86_64 psABI.
  742. This ABI is newer and uses 64-bits for data sizes and program pointers.
  743. The extra bits increase the footprint size of the programs, libraries,
  744. and also increases the memory and file system size requirements.
  745. Executing under the x32 psABI enables user programs to utilize CPU and system resources
  746. more efficiently while keeping the memory footprint of the applications low.
  747. Extra bits are used for registers but not for addressing mechanisms.
  748. </para>
  749. <section id='support'>
  750. <title>Support</title>
  751. <para>
  752. While the x32 psABI specifications are not fully finalized, this Yocto Project
  753. release supports current development specifications of x32 psABI.
  754. As of this release of the Yocto Project, x32 psABI support exists as follows:
  755. <itemizedlist>
  756. <listitem><para>You can create packages and images in x32 psABI format on x86_64 architecture targets.
  757. </para></listitem>
  758. <listitem><para>You can successfully build many recipes with the x32 toolchain.</para></listitem>
  759. <listitem><para>You can create and boot <filename>core-image-minimal</filename> and
  760. <filename>core-image-sato</filename> images.</para></listitem>
  761. </itemizedlist>
  762. </para>
  763. </section>
  764. <section id='stabilizing-and-completing-x32'>
  765. <title>Stabilizing and Completing x32</title>
  766. <para>
  767. As of this Yocto Project release, the x32 psABI kernel and library
  768. interfaces specifications are not finalized.
  769. </para>
  770. <para>
  771. Future Plans for the x32 psABI in the Yocto Project include the following:
  772. <itemizedlist>
  773. <listitem><para>Enhance and fix the few remaining recipes so they
  774. work with and support x32 toolchains.</para></listitem>
  775. <listitem><para>Enhance RPM Package Manager (RPM) support for x32 binaries.</para></listitem>
  776. <listitem><para>Support larger images.</para></listitem>
  777. </itemizedlist>
  778. </para>
  779. </section>
  780. <section id='using-x32-right-now'>
  781. <title>Using x32 Right Now</title>
  782. <para>
  783. Follow these steps to use the x32 spABI:
  784. <itemizedlist>
  785. <listitem><para>Enable the x32 psABI tuning file for <filename>x86_64</filename>
  786. machines by editing the <filename>conf/local.conf</filename> like this:
  787. <literallayout class='monospaced'>
  788. MACHINE = "qemux86-64"
  789. DEFAULTTUNE = "x86-64-x32"
  790. baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) \
  791. or 'INVALID'), True) or 'lib'}"
  792. #MACHINE = "genericx86"
  793. #DEFAULTTUNE = "core2-64-x32"
  794. </literallayout></para></listitem>
  795. <listitem><para>As usual, use BitBake to build an image that supports the x32 psABI.
  796. Here is an example:
  797. <literallayout class='monospaced'>
  798. $ bitbake core-image-sato
  799. </literallayout></para></listitem>
  800. <listitem><para>As usual, run your image using QEMU:
  801. <literallayout class='monospaced'>
  802. $ runqemu qemux86-64 core-image-sato
  803. </literallayout></para></listitem>
  804. </itemizedlist>
  805. </para>
  806. </section>
  807. </section>
  808. <section id="wayland">
  809. <title>Wayland</title>
  810. <para>
  811. <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Wayland</ulink>
  812. is a computer display server protocol that
  813. provides a method for compositing window managers to communicate
  814. directly with applications and video hardware and expects them to
  815. communicate with input hardware using other libraries.
  816. Using Wayland with supporting targets can result in better control
  817. over graphics frame rendering than an application might otherwise
  818. achieve.
  819. </para>
  820. <para>
  821. The Yocto Project provides the Wayland protocol libraries and the
  822. reference Weston compositor as part of its release.
  823. This section describes what you need to do to implement Wayland and
  824. use the compositor when building an image for a supporting target.
  825. </para>
  826. <section id="wayland-support">
  827. <title>Support</title>
  828. <para>
  829. The Wayland protocol libraries and the reference Weston compositor
  830. ship as integrated packages in the <filename>meta</filename> layer
  831. of the
  832. <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>.
  833. Specifically, you can find the recipes that build both Wayland
  834. and Weston at <filename>meta/recipes-graphics/wayland</filename>.
  835. </para>
  836. <para>
  837. You can build both the Wayland and Weston packages for use only
  838. with targets that accept the
  839. <ulink url='http://dri.freedesktop.org/wiki/'>Mesa 3D and Direct Rendering Infrastructure</ulink>,
  840. which is also known as Mesa DRI.
  841. This implies that you cannot build and use the packages if your
  842. target uses, for example, the
  843. <trademark class='registered'>Intel</trademark> Embedded Media and
  844. Graphics Driver (<trademark class='registered'>Intel</trademark>
  845. EMGD) that overrides Mesa DRI.
  846. </para>
  847. <note>
  848. Due to lack of EGL support, Weston 1.0.3 will not run directly on
  849. the emulated QEMU hardware.
  850. However, this version of Weston will run under X emulation without
  851. issues.
  852. </note>
  853. </section>
  854. <section id="enabling-wayland-in-an-image">
  855. <title>Enabling Wayland in an Image</title>
  856. <para>
  857. To enable Wayland, you need to enable it to be built and enable
  858. it to be included in the image.
  859. </para>
  860. <section id="enable-building">
  861. <title>Building</title>
  862. <para>
  863. To cause Mesa to build the <filename>wayland-egl</filename>
  864. platform and Weston to build Wayland with Kernel Mode
  865. Setting
  866. (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>)
  867. support, include the "wayland" flag in the
  868. <link linkend="var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></link>
  869. statement in your <filename>local.conf</filename> file:
  870. <literallayout class='monospaced'>
  871. DISTRO_FEATURES_append = " wayland"
  872. </literallayout>
  873. </para>
  874. <note>
  875. If X11 has been enabled elsewhere, Weston will build Wayland
  876. with X11 support
  877. </note>
  878. </section>
  879. <section id="enable-installation-in-an-image">
  880. <title>Installing</title>
  881. <para>
  882. To install the Wayland feature into an image, you must
  883. include the following
  884. <link linkend='var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></link>
  885. statement in your <filename>local.conf</filename> file:
  886. <literallayout class='monospaced'>
  887. CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
  888. </literallayout>
  889. </para>
  890. </section>
  891. </section>
  892. <section id="running-weston">
  893. <title>Running Weston</title>
  894. <para>
  895. To run Weston inside X11, enabling it as described earlier and
  896. building a Sato image is sufficient.
  897. If you are running your image under Sato, a Weston Launcher appears
  898. in the "Utility" category.
  899. </para>
  900. <para>
  901. Alternatively, you can run Weston through the command-line
  902. interpretor (CLI), which is better suited for development work.
  903. To run Weston under the CLI, you need to do the following after
  904. your image is built:
  905. <orderedlist>
  906. <listitem><para>Run these commands to export
  907. <filename>XDG_RUNTIME_DIR</filename>:
  908. <literallayout class='monospaced'>
  909. mkdir -p /tmp/$USER-weston
  910. chmod 0700 /tmp/$USER-weston
  911. export XDG_RUNTIME_DIR=/tmp/$USER=weston
  912. </literallayout></para></listitem>
  913. <listitem><para>Launch Weston in the shell:
  914. <literallayout class='monospaced'>
  915. weston
  916. </literallayout></para></listitem>
  917. </orderedlist>
  918. </para>
  919. </section>
  920. </section>
  921. <section id="licenses">
  922. <title>Licenses</title>
  923. <para>
  924. This section describes the mechanism by which the OpenEmbedded build system
  925. tracks changes to licensing text.
  926. The section also describes how to enable commercially licensed recipes,
  927. which by default are disabled.
  928. </para>
  929. <para>
  930. For information that can help you maintain compliance with various open
  931. source licensing during the lifecycle of the product, see the
  932. "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Project's Lifecycle</ulink>" section
  933. in the Yocto Project Development Manual.
  934. </para>
  935. <section id="usingpoky-configuring-LIC_FILES_CHKSUM">
  936. <title>Tracking License Changes</title>
  937. <para>
  938. The license of an upstream project might change in the future.
  939. In order to prevent these changes going unnoticed, the
  940. <filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename>
  941. variable tracks changes to the license text. The checksums are validated at the end of the
  942. configure step, and if the checksums do not match, the build will fail.
  943. </para>
  944. <section id="usingpoky-specifying-LIC_FILES_CHKSUM">
  945. <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>
  946. <para>
  947. The <filename>LIC_FILES_CHKSUM</filename>
  948. variable contains checksums of the license text in the source code for the recipe.
  949. Following is an example of how to specify <filename>LIC_FILES_CHKSUM</filename>:
  950. <literallayout class='monospaced'>
  951. LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
  952. file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
  953. file://licfile2.txt;endline=50;md5=zzzz \
  954. ..."
  955. </literallayout>
  956. </para>
  957. <para>
  958. The build system uses the
  959. <filename><link linkend='var-S'>S</link></filename> variable as
  960. the default directory used when searching files listed in
  961. <filename>LIC_FILES_CHKSUM</filename>.
  962. The previous example employs the default directory.
  963. </para>
  964. <para>
  965. Consider this next example:
  966. <literallayout class='monospaced'>
  967. LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
  968. md5=bb14ed3c4cda583abc85401304b5cd4e"
  969. LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
  970. </literallayout>
  971. </para>
  972. <para>
  973. The first line locates a file in
  974. <filename>${S}/src/ls.c</filename>.
  975. The second line refers to a file in
  976. <filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>.
  977. </para>
  978. <para>
  979. Note that <filename>LIC_FILES_CHKSUM</filename> variable is
  980. mandatory for all recipes, unless the
  981. <filename>LICENSE</filename> variable is set to "CLOSED".
  982. </para>
  983. </section>
  984. <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax">
  985. <title>Explanation of Syntax</title>
  986. <para>
  987. As mentioned in the previous section, the
  988. <filename>LIC_FILES_CHKSUM</filename> variable lists all the
  989. important files that contain the license text for the source code.
  990. It is possible to specify a checksum for an entire file, or a specific section of a
  991. file (specified by beginning and ending line numbers with the "beginline" and "endline"
  992. parameters, respectively).
  993. The latter is useful for source files with a license notice header,
  994. README documents, and so forth.
  995. If you do not use the "beginline" parameter, then it is assumed that the text begins on the
  996. first line of the file.
  997. Similarly, if you do not use the "endline" parameter, it is assumed that the license text
  998. ends with the last line of the file.
  999. </para>
  1000. <para>
  1001. The "md5" parameter stores the md5 checksum of the license text.
  1002. If the license text changes in any way as compared to this parameter
  1003. then a mismatch occurs.
  1004. This mismatch triggers a build failure and notifies the developer.
  1005. Notification allows the developer to review and address the license text changes.
  1006. Also note that if a mismatch occurs during the build, the correct md5
  1007. checksum is placed in the build log and can be easily copied to the recipe.
  1008. </para>
  1009. <para>
  1010. There is no limit to how many files you can specify using the
  1011. <filename>LIC_FILES_CHKSUM</filename> variable.
  1012. Generally, however, every project requires a few specifications for license tracking.
  1013. Many projects have a "COPYING" file that stores the license information for all the source
  1014. code files.
  1015. This practice allows you to just track the "COPYING" file as long as it is kept up to date.
  1016. </para>
  1017. <tip>
  1018. If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match
  1019. error and displays the correct "md5" parameter value during the build.
  1020. The correct parameter is also captured in the build log.
  1021. </tip>
  1022. <tip>
  1023. If the whole file contains only license text, you do not need to use the "beginline" and
  1024. "endline" parameters.
  1025. </tip>
  1026. </section>
  1027. </section>
  1028. <section id="enabling-commercially-licensed-recipes">
  1029. <title>Enabling Commercially Licensed Recipes</title>
  1030. <para>
  1031. By default, the OpenEmbedded build system disables
  1032. components that have commercial or other special licensing
  1033. requirements.
  1034. Such requirements are defined on a
  1035. recipe-by-recipe basis through the <filename>LICENSE_FLAGS</filename> variable
  1036. definition in the affected recipe.
  1037. For instance, the
  1038. <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
  1039. recipe contains the following statement:
  1040. <literallayout class='monospaced'>
  1041. LICENSE_FLAGS = "commercial"
  1042. </literallayout>
  1043. Here is a slightly more complicated example that contains both an
  1044. explicit recipe name and version (after variable expansion):
  1045. <literallayout class='monospaced'>
  1046. LICENSE_FLAGS = "license_${PN}_${PV}"
  1047. </literallayout>
  1048. In order for a component restricted by a <filename>LICENSE_FLAGS</filename>
  1049. definition to be enabled and included in an image, it
  1050. needs to have a matching entry in the global
  1051. <filename>LICENSE_FLAGS_WHITELIST</filename> variable, which is a variable
  1052. typically defined in your <filename>local.conf</filename> file.
  1053. For example, to enable
  1054. the <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename>
  1055. package, you could add either the string
  1056. "commercial_gst-plugins-ugly" or the more general string
  1057. "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>.
  1058. See the
  1059. "<link linkend='license-flag-matching'>License Flag Matching</link>" section
  1060. for a full explanation of how <filename>LICENSE_FLAGS</filename> matching works.
  1061. Here is the example:
  1062. <literallayout class='monospaced'>
  1063. LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
  1064. </literallayout>
  1065. Likewise, to additionally enable the package built from the recipe containing
  1066. <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, and assuming
  1067. that the actual recipe name was <filename>emgd_1.10.bb</filename>,
  1068. the following string would enable that package as well as
  1069. the original <filename>gst-plugins-ugly</filename> package:
  1070. <literallayout class='monospaced'>
  1071. LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
  1072. </literallayout>
  1073. As a convenience, you do not need to specify the complete license string
  1074. in the whitelist for every package.
  1075. You can use an abbreviated form, which consists
  1076. of just the first portion or portions of the license string before
  1077. the initial underscore character or characters.
  1078. A partial string will match
  1079. any license that contains the given string as the first
  1080. portion of its license.
  1081. For example, the following
  1082. whitelist string will also match both of the packages
  1083. previously mentioned as well as any other packages that have
  1084. licenses starting with "commercial" or "license".
  1085. <literallayout class='monospaced'>
  1086. LICENSE_FLAGS_WHITELIST = "commercial license"
  1087. </literallayout>
  1088. </para>
  1089. <section id="license-flag-matching">
  1090. <title>License Flag Matching</title>
  1091. <para>
  1092. License flag matching allows you to control what recipes the
  1093. OpenEmbedded build system includes in the build.
  1094. Fundamentally, the build system attempts to match
  1095. <filename>LICENSE_FLAGS</filename> strings found in
  1096. recipes against <filename>LICENSE_FLAGS_WHITELIST</filename>
  1097. strings found in the whitelist.
  1098. A match causes the build system to include a recipe in the
  1099. build, while failure to find a match causes the build system to
  1100. exclude a recipe.
  1101. </para>
  1102. <para>
  1103. In general, license flag matching is simple.
  1104. However, understanding some concepts will help you
  1105. correctly and effectively use matching.
  1106. </para>
  1107. <para>
  1108. Before a flag
  1109. defined by a particular recipe is tested against the
  1110. contents of the whitelist, the expanded string
  1111. <filename>_${PN}</filename> is appended to the flag.
  1112. This expansion makes each <filename>LICENSE_FLAGS</filename>
  1113. value recipe-specific.
  1114. After expansion, the string is then matched against the
  1115. whitelist.
  1116. Thus, specifying
  1117. <filename>LICENSE_FLAGS = "commercial"</filename>
  1118. in recipe "foo", for example, results in the string
  1119. <filename>"commercial_foo"</filename>.
  1120. And, to create a match, that string must appear in the
  1121. whitelist.
  1122. </para>
  1123. <para>
  1124. Judicious use of the <filename>LICENSE_FLAGS</filename>
  1125. strings and the contents of the
  1126. <filename>LICENSE_FLAGS_WHITELIST</filename> variable
  1127. allows you a lot of flexibility for including or excluding
  1128. recipes based on licensing.
  1129. For example, you can broaden the matching capabilities by
  1130. using license flags string subsets in the whitelist.
  1131. <note>When using a string subset, be sure to use the part of
  1132. the expanded string that precedes the appended underscore
  1133. character (e.g. <filename>usethispart_1.3</filename>,
  1134. <filename>usethispart_1.4</filename>, and so forth).
  1135. </note>
  1136. For example, simply specifying the string "commercial" in
  1137. the whitelist matches any expanded
  1138. <filename>LICENSE_FLAGS</filename> definition that starts with
  1139. the string "commercial" such as "commercial_foo" and
  1140. "commercial_bar", which are the strings the build system
  1141. automatically generates for hypothetical recipes named
  1142. "foo" and "bar" assuming those recipes simply specify the
  1143. following:
  1144. <literallayout class='monospaced'>
  1145. LICENSE_FLAGS = "commercial"
  1146. </literallayout>
  1147. Thus, you can choose to exhaustively
  1148. enumerate each license flag in the whitelist and
  1149. allow only specific recipes into the image, or
  1150. you can use a string subset that causes a broader range of
  1151. matches to allow a range of recipes into the image.
  1152. </para>
  1153. <para>
  1154. This scheme works even if the
  1155. <filename>LICENSE_FLAGS</filename> string already
  1156. has <filename>_${PN}</filename> appended.
  1157. For example, the build system turns the license flag
  1158. "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would
  1159. match both the general "commercial" and the specific
  1160. "commercial_1.2_foo" strings found in the whitelist, as
  1161. expected.
  1162. </para>
  1163. <para>
  1164. Here are some other scenarios:
  1165. <itemizedlist>
  1166. <listitem><para>You can specify a versioned string in the
  1167. recipe such as "commercial_foo_1.2" in a "foo" recipe.
  1168. The build system expands this string to
  1169. "commercial_foo_1.2_foo".
  1170. Combine this license flag with a whitelist that has
  1171. the string "commercial" and you match the flag along
  1172. with any other flag that starts with the string
  1173. "commercial".</para></listitem>
  1174. <listitem><para>Under the same circumstances, you can
  1175. use "commercial_foo" in the whitelist and the
  1176. build system not only matches "commercial_foo_1.2" but
  1177. also matches any license flag with the string
  1178. "commercial_foo", regardless of the version.
  1179. </para></listitem>
  1180. <listitem><para>You can be very specific and use both the
  1181. package and version parts in the whitelist (e.g.
  1182. "commercial_foo_1.2") to specifically match a
  1183. versioned recipe.</para></listitem>
  1184. </itemizedlist>
  1185. </para>
  1186. </section>
  1187. <section id="other-variables-related-to-commercial-licenses">
  1188. <title>Other Variables Related to Commercial Licenses</title>
  1189. <para>
  1190. Other helpful variables related to commercial
  1191. license handling exist and are defined in the
  1192. <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file:
  1193. <literallayout class='monospaced'>
  1194. COMMERCIAL_AUDIO_PLUGINS ?= ""
  1195. COMMERCIAL_VIDEO_PLUGINS ?= ""
  1196. COMMERCIAL_QT = ""
  1197. </literallayout>
  1198. If you want to enable these components, you can do so by making sure you have
  1199. statements similar to the following
  1200. in your <filename>local.conf</filename> configuration file:
  1201. <literallayout class='monospaced'>
  1202. COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
  1203. gst-plugins-ugly-mpegaudioparse"
  1204. COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
  1205. gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
  1206. COMMERCIAL_QT ?= "qmmp"
  1207. LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
  1208. </literallayout>
  1209. Of course, you could also create a matching whitelist
  1210. for those components using the more general "commercial"
  1211. in the whitelist, but that would also enable all the
  1212. other packages with <filename>LICENSE_FLAGS</filename> containing
  1213. "commercial", which you may or may not want:
  1214. <literallayout class='monospaced'>
  1215. LICENSE_FLAGS_WHITELIST = "commercial"
  1216. </literallayout>
  1217. </para>
  1218. <para>
  1219. Specifying audio and video plug-ins as part of the
  1220. <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and
  1221. <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements
  1222. or commercial Qt components as part of
  1223. the <filename>COMMERCIAL_QT</filename> statement (along
  1224. with the enabling <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the
  1225. plug-ins or components into built images, thus adding
  1226. support for media formats or components.
  1227. </para>
  1228. </section>
  1229. </section>
  1230. </section>
  1231. </chapter>
  1232. <!--
  1233. vim: expandtab tw=80 ts=4
  1234. -->