[GUIDE] Android Repo Mirroring - Android Software Development

I assume you already have all your build packages installed. This guide is merely showing how to obtain a mirror copy of the android source for local purposes.
I will be using OmniRom for the duration of this guide, but for those who use CM or other projects, the concept should remain the same.
I do not expect this guide to be useful for everyone but there will be many who are going to be thankful that did not initially know this could be done.
Due to the rapid development, I ran into a huge issue pretty quickly when having to nuke/resync the various repos for building. The total space required for a full omni source is roughly 508,334 items, totaling 14.4 GB based on today from branch android-5.1 and requires pulling from a massive number of git repos online. So if bandwidth is an issue then constantly syncing after nuking the source could become an issue for you real quick. Not to mention the time it takes just to pull all of this data from the internet onto your computer.
However, by creating a Mirror of the source and its numerous git repos, the source is roughly 19,696 items, totalling 16.5 GB based on today from branch android-5.1.
Now, for those asking, why do a mirror? It is relatively simple. When using several sources, especially in situations where bandwidth is scarce, it is better to create a local mirror of the entire content, and to sync from that mirror (which requires no network access). The download for a full mirror is smaller than the download of two sources, while containing more information.
These instructions assume that the mirror is created in /path/to/home/android/mirror. The first step is to create and sync the mirror itself.
Code:
mkdir -p ~/android/mirror
Code:
cd ~/android/mirror
Code:
repo init -u https://github.com/omnirom/android.git -b android-5.1 [COLOR="Red"]--mirror[/COLOR]
Code:
repo sync -j# -f --no-clone-bundle
Basically this allows you to initialize a local mirror of an android repo (--mirror) and then later allow you to point subsequent repo pulls to that local copy which will be explained further into this guide. The mirror looks nothing like the actual source used to make builds.
Once the mirror repo has synced, then we are ready to establish the actual source in another directory that will pull directly from the mirror locally.
Code:
mkdir -p ~/android/omni
Code:
cd ~/android/omni
Code:
repo init -u git://github.com/omnirom/android.git -b android-5.1 [COLOR="Red"]--reference=/path/to/home/android/mirror/[/COLOR]
Code:
repo sync -j# -f --no-clone-bundle
Note the use of the ‘–-reference’ parameter. This uses my local mirror of the android sources instead of pulling across the network again.
Now I can pull one of the various repos, using that same ‘--reference’ parameter (pointed at my local omni mirror) and profit!
From here on out, should I want to update the omni source with the latest changes I just go to the mirror directory and run:
Code:
repo sync
Then, I switch over to the omni directory and run the same command to pull those changes from the local directory.
Even further, should I nuke something in the omni directory, I can simply delete it and run repo sync to pull from the local mirror to fix the problem and never have to worry about pulling data from the network itself.
Here is an example for a clearer picture:
I make changes in /android/omni/bootable/recovery
I do not like the changes I have made
I delete /android/omni/bootable recursively
I run repo sync to restore /android/omni/bootable
The sync pulls from the local mirror instead of downloading the repo from the internet.
Time for syncing is cut in half if not more
Everything is still intact

Thanks for that bro us Linux noobs will find it very useful

Related

[REF] Compiling CM6 for Milestone XT720

First, follow these instructions to get your system setup for build:
http://wiki.cyanogenmod.com/index.php?title=Compile_CyanogenMod_for_Sholes
You'll need at least 15GB of disk space. Once you've got all the prerequisites installed:
Code:
mkdir ~/android
cd ~/android
repo init -u git://github.com/Mioze7Ae/android.git -b froyo
repo sync -j 2
. build/envsetup.sh
lunch cyanogen_sholest-eng
vendor/cyanogen/get-rommanager
make -j 2 bacon
If everything goes well, the update package will be something like out/target/product/sholest/update-cm-6.1.2-0.10-11.08.08-MilestoneXT720-signed.zip
This is my current understanding of the voodoo:
repo init : sets up your repository and downloads a manifest file that identifies other repositories
repo sync : downloads all the repositories and checks out a working copy
. build/envsetup.sh : parses all the makefiles and such to create a menu of available devices to build
lunch : configures the source tree to build a specific device (run lunch with no arguments to get a menu)
get-rommanager : apparently needed for some reason
make bacon : actually builds everything and creates the ota package. You may want to consider something like "make -j 8 bacon >build.out 2>build.err"
repo sync takes either metric or imperial eons (whichever is larger) (-j N can help by opening multiple concurrent downloads if you have a fast uplink). On a dual AMD Opteron 248 machine with 6GB build takes about 3 hours (Ubuntu 11.04 64-bit). On a dual Xeon [email protected] (8 cores total) with 16GB RAM running debian lenny 64-bit, build takes ~15 min. You want to pick the -j to be 1-2 times the number of CPUs (I used -j 2 on the AMD and -j 8 on the Xeon). The android build system does a pretty good job at pegging your CPUs.
I don't currently know how to tag the state of the entire tree, so when nadlabak or milaq or anyone else updates repositories that I didn't fork, there's a chance it may break things from time-to-time. I'm currently waiting for a build from a fresh checkout to complete. Hopefully it's not broken...
Edit: Fresh compile succeeded, update applied and booted. Happy hacking!
Feel free to peruse my repositories at http://github.com/Mioze7Ae
See also: http://android.doshaska.net/cm6build
Thanks: nadlabak, milaq, stlifey, #milestone-modding
Thanks, will add cm6 to mine build server. =)
Maybe we should make todo list and make changes that we need?
Then move it to main build tree?
fjfalcon said:
Thanks, will add cm6 to mine build server. =)
Maybe we should make todo list and make changes that we need?
Then move it to main build tree?
Click to expand...
Click to collapse
Im on it.. flashing once i have finished uploading XT720 XDA Special Hellmonger Edition, il collect data and proc
i was able to compile configs.ko
so we can dump the config of the running moto kernel
Code:
insmod configs.ko
zcat /proc/config.gz
attached is tar with module and dumped config from Central Europe kernel...
so may be finally we will be able to "debug" why on the singapore kernel it is stated that tvout works, and did not work on other kernels....
So please do:
Code:
adb push configs.ko /tmp
adb shell
/system/bin/insmod /tmp/configs.ko
exit from adb shell
Code:
adb pull /proc/config.gz
and post
Which Kernel (e.g. Central European, Korean, Singapore, Cincinaty etc....) is used
and the corresponding config.gz
edit: forget to attache
badly.... it is not the runing kernel config, but the config i have compiled.... sorry
du -h .repo - how much it has?
fjfalcon said:
du -h .repo - how much it has?
Click to expand...
Click to collapse
$ du -hs .repo
4.2G .repo
$ du -hs .
12G .
FYI, I just merged updates from milaq's repo and broke the build. Will update when I get it sorted again.
Mioze7Ae said:
FYI, I just merged updates from milaq's repo and broke the build. Will update when I get it sorted again.
Click to expand...
Click to collapse
It's working again. I'm still getting the hang of this repo/git stuff. What appears to have happened is that when I first forked milaq's repo a few weeks ago I happened to fork after the first part of a two-part commit. So, now the tree is sync'd up with milaq's latest (0.12--screenshot in powermenu, w00t!). It installs and boots but it's minimally tested.
Maybe you can archive android dir and paste it to some speed hosting?
Cause i still getting it at build server at mine work that currently free....
fjfalcon said:
Maybe you can archive android dir and paste it to some speed hosting?
Cause i still getting it at build server at mine work that currently free....
Click to expand...
Click to collapse
Hmm. I'm I'll see about that, but .repo is 4GB... and android is 12GB... that's going to be slow nomatter what... and those stupid rapid/multi/whatevers don't seem to handle resume... after the initial repo sync, the next sync is much faster.
Is repo sync still down? Or rather, is android.git.kernel.org still down? I tried repo syncing from RC2 to RC3, so I could implement a patch for Exchange security features (more on this later, I will update when I get it working).
For now, I will try the compile on RC2.
I do know it was down last Friday, but I haven't had a chance to try more recently.

[NEED HELP] There are some problems in using "config.sh"

Hi,
I'm going to port Firefox OS to my I9082 but I have some problems after "./config.sh i9082".
It can't get some repositories from android.git.linaro.com and after that it stuck at "Fetching objects: 25% (20/78)" and after this, I waited for 8 hours!!!!! :crying:
after this, i closed the terminal.
I tried again using "repo sync" to continue but there is a new problem! It says that some of repos not found! for example i can't get "platform/external/freetype" and "platform/external/dbus" from linaro git.
Can anybody help me?
Delete broken projects and sync again
DataUnit said:
Hi,
I'm going to port Firefox OS to my I9082 but I have some problems after "./config.sh i9082".
It can't get some repositories from android.git.linaro.com and after that it stuck at "Fetching objects: 25% (20/78)" and after this, I waited for 8 hours!!!!! :crying:
after this, i closed the terminal.
I tried again using "repo sync" to continue but there is a new problem! It says that some of repos not found! for example i can't get "platform/external/freetype" and "platform/external/dbus" from linaro git.
Can anybody help me?
Click to expand...
Click to collapse
Delete all folder expect .repo and start over.
arky_r said:
Delete all folder expect .repo and start over.
Click to expand...
Click to collapse
I did this action 4 times and when I executed the command again, it will stop at another project! :crying:
DataUnit said:
I did this action 4 times and when I executed the command again, it will stop at another project! :crying:
Click to expand...
Click to collapse
Did you reading the repo tips at xda-universityDOTcom ?
Syncing a full new source tree in minutes!
Time might come when you’ll need to work with several full source trees. For example, you may want to have an AOKP source tree as well as a CM tree.
Obvisouly, you could repeat the same steps, to repo init and then repo sync -j 8 -c for both. But that would be twice as time consuming.
There is an alternative that should be much faster. Lets say that you have already initialized and synced a CM source tree, in ~/android/cm.
Now, we will prepare a new repo for AOKP tree, in ~/android/aokp. Here comes the trick: We will use the --reference flag of repo init to indicate to repo that we already have a local repository that it can take information from, instead of downloading everything from Internet again:
mkdir ~/android/aokp cd ~/android/aokp repo init --reference=~/android/cm -u git://github.com/AOKP/platform_manifest.git -b jb
1
2
3
mkdir ~/android/aokp
cd ~/android/aokp
repo init --reference=~/android/cm -u git://github.com/AOKP/platform_manifest.git -b jb
You should notice a big time diference, from hours for a full repo sync, down to minutes using this trick!

Using Git in an unusual workflow

Hi people, this is my first post here, I think is the best place where I can discuss my programming issues. If this post isn't in the correct place, please move it and forgive my dumbness.
I'm working in some personal projects, android apps, in my house. There I have my PC with Eclipse and Git, at the end of the weekend I'm always commiting changes and uploading them to GitHub. When I go to my office, sometimes I have some spare time and I open Eclipse to start working, but of course I synchronize the GitHub repository data with my local data (in order to work with the latest changes I've made in my house), but I'm always having troubles with this action: I don't know how to discard my local old files, keep the files that haven't been modified and download the latest version of my modified files. I always end deleting the folder and cloning the repo again because I mess the files... I'm sure there is a better way.
So, I'd be delighted if you could assist me on how should I work with this feature of Git. I'd like too to take advice to improve my workflow. I've checked lots of sites and tutorials about Git but I haven't been able to get useful info to match with my workflow...
Thank you in advance
kyomuga said:
Hi people, this is my first post here, I think is the best place where I can discuss my programming issues. If this post isn't in the correct place, please move it and forgive my dumbness.
I'm working in some personal projects, android apps, in my house. There I have my PC with Eclipse and Git, at the end of the weekend I'm always commiting changes and uploading them to GitHub. When I go to my office, sometimes I have some spare time and I open Eclipse to start working, but of course I synchronize the GitHub repository data with my local data (in order to work with the latest changes I've made in my house), but I'm always having troubles with this action: I don't know how to discard my local old files, keep the files that haven't been modified and download the latest version of my modified files. I always end deleting the folder and cloning the repo again because I mess the files... I'm sure there is a better way.
So, I'd be delighted if you could assist me on how should I work with this feature of Git. I'd like too to take advice to improve my workflow. I've checked lots of sites and tutorials about Git but I haven't been able to get useful info to match with my workflow...
Thank you in advance
Click to expand...
Click to collapse
Hi, this should be the right section for talking git, don't worry
My understanding is that you have some local (uncommited) changes on your machine at work, and you want to pull the latest commits from your distant repository while avoiding merge conflicts, is that correct ?
If it is, then you have several solutions at your disposal:
The easiest and cleanest way is to stash your local changes, pull the new commits and then pop the stashed changes to commit them over the up-to-date repo later on:
(replace <remote> and <branch> with your actual remote and target branch, for example git pull origin master)
Code:
git stash
git pull <remote> <branch>
git stash pop
The branch is now up to date and your local changes are still there for you to commit or to continue modifying.
If instead you decide that you don't need the local changes you stashed anymore, instead of popping them you can clear the stash:
Code:
git stash clear
If you want to fetch remote changes for all the branches, replace the git pull with:
Code:
git fetch <remote>
If all you want is to get rid of your local changes and sync with upstream repo, you can use:
Code:
git checkout .
git pull <remote> <branch>
Hope this helps, let me know if you need further help, or if I didn't understand your question properly :silly:
Androguide.fr said:
If all you want is to get rid of your local changes and sync with upstream repo, you can use:
Code:
git checkout .
git pull <remote> <branch>
Click to expand...
Click to collapse
Thank you very much. This is what I need, or at least what fits best with my workflow, as I'm always starting programming from the point i've stopped working in another place, being like this:
When I get to work:
Code:
git checkout .
git pull <remote> <branch>
So I'm getting my updated code as I stopped programming at home
Minutes after going home:
Code:
git add .
git commit -m "another day, another commit"
git push origin master
When I get home:
Code:
git checkout .
git pull <remote> <branch>
So I'm getting my updated code as I stopped programming at work
Minutes after going to bed:
Code:
git add .
git commit -m "another day, another commit"
git push origin master
This is a correct workflow? or I'm misusing the benefits of Git?
kyomuga said:
Thank you very much. This is what I need, or at least what fits best with my workflow, as I'm always starting programming from the point i've stopped working in another place, being like this:
When I get to work:
Code:
git checkout .
git pull <remote> <branch>
So I'm getting my updated code as I stopped programming at home
Minutes after going home:
Code:
git add .
git commit -m "another day, another commit"
git push origin master
When I get home:
Code:
git checkout .
git pull <remote> <branch>
So I'm getting my updated code as I stopped programming at work
Minutes after going to bed:
Code:
git add .
git commit -m "another day, another commit"
git push origin master
This is a correct workflow? or I'm misusing the benefits of Git?
Click to expand...
Click to collapse
Yeah, this looks correct, as long as you don't need your local changes this is probably the best way.

Repo Download Size of CM-14.1 Lineage Branch?

Need some advice here, thought I'd give building Lineage from source a go as it would open up a whole lot of opportunities from there. But I didn't think it would require a massive (unknown size) download to even begin. Could anyone give me an idea of realistically what a repo sync such as "repo init -u https://github.com/LineageOS/android.git -b cm-14.1" would make me download.?
Unfortunately I'm an aussie and data caps are quite strict here, so I can't really go out downloading massive files of unknown sizes.

[GUIDE] HOW TO BUILD ANDROID FROM SOURCE [2020 Edition]

[NOOB-FRIENDLY] HOW TO BUILD ANDROID FROM SOURCE CODE [2020 EDITION]
Introduction
Hi, my name is Varun, during the times I was building I faced some trouble building, and I thought it would be helpful if I put together a guide that covers every aspect of building. Many roms fundamentally differ so this is a generic guide, but when it comes to modifying device specific files you are going to have to refer to your rom developer or your device tree maintainer.
I see a lot of lead devs referring to older guides on xda, and felt like there should be a good reference thats more recent.
I have heavily depended upon Nathan Chance's Guide which can be found HERE. I feel like this may be a little intimidating for new builders hence this thread. I will slowly upload screenshots here as I find time. But for now this is what I have. You can find my guide HERE. If there are any changes to be made in this thread then it also needs to be changed on gh, so please do comment or make a pr and fix what is requested.
Table of Contents
1) Getting started
2) Prerequisites
3) Understanding the DOC
4) Preparing Linux environment
5) Getting the source
6) Getting device specific files
7) Building
GETTING STARTED
• This guide will help you understand the basics of rom building.
• You must have exprience with linux and terminal
• You must read everything at least once
• If you get stuck at any step, google is your best friend.
If you still struggle, these are places you can get help
1) This thread
2) GitHub
3) Other XDA threads
4) Telegram (Your SOC groups or Android Builders Help)
• Build only if you have time to spare, never rush through the learning stage
• Never think about skipping a step
• If you are planning to modify a ROM, always ask for permission
• If you are planning on including other peoples work in your rom always ask for permission
Prerequisites
• Your system should have at least 8GB RAM and 4 cores (Use Zram for 8gb ram devices)
• Strong internet connection. You will be downloading 50GB+ worth of files
• Ubuntu - this should be installed alongside a primary OS. NOT VM
• Minimum 250GB of free disk space (you can sync 1 ROMs and build for 3 devices at once)
• Git - Read some documentation
• Command Line - get yourself familiar, learn with CodeAcademy
Understanding This Doc
• Notes with background information will be given in italics
•
Code:
is used to represent a command that is run from terminal
PREPARING LINUX ENVIRONMENT FOR BUILDING
• Now the you have finished setting up, a special environment is required for building.
• Preparing the environment is as easy as double clicking .exe in Windows or .dmg in MacOS
thanks to developer Akhil N.
• We need to clone his scripts (installer) from his GitHub
NOTE:
A script is a file which contains multiple commands that run sequentially when executed
Code:
git clone https://github.com/akhilnarang/scripts
• This should copy the scripts to directory /home/username/scripts
• Cd into the directory called setup
Code:
cd scripts/setup
• List out directory
Code:
ls
• Find the file that corresponds to our Linux build. Since we are using Ubuntu it will be android_build_env.sh
For other Distros refer to the readme that has also been cloned
• Run the script
Code:
. android_build_env.sh
• Setup GitHub using on screen instructions
• Congratulations you set up your computer for building
• Go back the previous dir that you were originally on (/home/username)
Code:
cd
NOTE:
cd will always take you to home/username
GETTING THE SOURCE
• This is probably one of the worst steps, you have to wait a while
• GCloud users will be able to download source in 15 minutes
• Other users may have to wait longer (depends on internet connection speed)
• Make a directory for your ROM
Code:
mkdir “ROMNAME”
NOTE:
this will make folder in directory home/username/romname
• Cd in and initiate repository, this basically tells your computer where to download source files from.
The repository initiation command can be found on the GitHub page under “manifest” then scroll down to
repo initialisation and copy command
• It should look like
Code:
repo init -u git://github.com/crdroidandroid/android.git -b 10.0
HINTS
NOTE:
1) Linux users can paste into terminal using Ctrl + Shift + V
2) Linux users can copy from terminal using Ctrl + Shift + C
3) GCloud users can paste into window using Ctrl + V
4) GCloud users can copy from window simply by highlighting
5) GCloud users using ssh from terminal can just follow terminal commands
• Once repo is initialised, you can begin download using
Code:
repo sync -f --force-sync --no-clone-bundle --no-tags -j$(nproc --all)
• Let everything download
GETTING DEVICE SPECIFIC FILES
• As most of you know the kernel acts as the translator between the OS and your actual hardware
• The files mentioned above are android version specific and will only work with that android
version
• The developer for your device aka dt maintainer will have these files on their GitHub and its
usually linked at the end of the op of other rom threads in the source code section
• You have to make sure that the rom version you downloaded matches up
• If the device tree is for an older version then wait until your dev. releases the latest trees, if no
developer is working on your device then it is possible to modify the tree to work with newer
version of android but it may take some effort.
• In case your tree doesnt match with the rom version that you have downloaded, the easiest and best option is
to scrap everything and download the source version that your tree is designed for
• To scrap everything, use
Code:
rm -rf “folder containing rom"
• To modify the init command to download an older version you need to pick out the branch you
need (the branch is basically the version the newer version is added to a new branch)
• Change the init command after -b to include desired branch. For example
repo init -b ten to repo init -b seven
• To get the device specific files, we need to fork them to your GitHub.
Find the stuff (kernel vendor device tree hardware) and fork them
NOTE:
Forking basically copies them to your GitHub
• At this point we need to know if we need to modify the device tree as some ROMs don’t require
you to modify it (CrDroid)(LOS)
• Some roms require only partial modifications, this is rom specific so I wont be going too much into detail
• If modification is required follow these steps , otherwise skip to the end of this section
• After forking we need to modify the files we will do this from the browser for ease of
understanding
•There are three files that need modification and they are located your device tree
• Some tree developers may split your device tree into two parts if there is more than one phone
running the same SOC, so keep an eye out for that. (devices that have the same soc are like the op3/op3t and the op7/op7p)
• The files that we need to modify are under your "device codename" device tree, not "device SOC-common" device tree.
• For example; we need to go to "device_samsung_crownlte" tree not "device_samsung_universal9810-common"
• In here the three files that need modification are
1) Android Products.mk
2) lineage.dependencies
3) lineage_"device codename".mk (example:lineage_crownlte.mk)
• First open AndriodProducts.mk Here replace wherever you see lineage with name of ROM in lowercase only.
• Second, open lineage dependencies.mk and rename file to "romname".dependencies (ecample: aim.dependencies)
• Third, open lineage_"codename".mk and rename to "romname"_"codename".mk and replace
lineage with "romname in the code. (Remember,use lowercase)
NOTE:
If you are building an older version of android make sure you branch out the right branch for your trees
People who do not need to modify can join us here
• Now that all your files are in your GitHub, we need to download them
• There are two easy ways
1) Fast and easy Room Service
2) Slow and boring manual cloning
• Room Service
1) cd into .repo from romsource
Code:
cd .repo
2) Make directory local_manifests
Code:
mkdir local_manifests
3) Make room service file
Code:
nano roomservice.xml
(follow next bullet point on how to make roomservice.xml)
4) Return to ROM folder
Code:
cd && cd "romfolder"
• Making the roomservice.xml is as easy as copy paste.
Visit
https://github.com/vrnsoma/local_manifests/blob/lineage-17.1/roomservice.xml
and copy and paste it into a text editor and edit it then paste it into the terminal window (step 3 in above point)
Hint for Device Tree
Example: link= userabc/repositories/device_samsung/crownlte (location on github)
path= device/samsung/crownlte (Where it should download to)
remote= github (where files are stored)
revision= branchname (which version you want)
• Resync Repo like I mentioned before
Code:
repo sync -f --force-sync --no-clone-bundle --no-tags -j$(nproc --all)
BUILDING
• Run
Code:
. build/envsetup.sh
• Then
Code:
brunch codename
• Make sure to clean build by running this in between builds
Code:
make clobber && make clean
• If you are using a slow system and want to improve build speeds, use the following command
for making partially clean builds for incremental updates
Code:
make installclean
• Building without cleaning is called dirty building and is never advised and do not do this unless
you know what you are doing
KEEPING SETUP UP TO DATE
• Repo sync will keep your rom and device specific files that have been deined in roomservice up to date
CREDITS
NathanChance
All threads on XDA
All guides on Github
All devs found on Telegram
If you want specific credits plz do notify me!
XDA:DevDB Information
BUILDING ANDROID, ROM for all devices (see above for details)
Contributors
vsoma
ROM OS Version: Android 10
ROM Firmware Required: NOOB
Version Information
Status: Stable
Created 2020-10-03
Last Updated 2020-10-03
I need rom porting guide for android 10....can you make thread on it?
I tried the tutorial, after the step repo sync -f --force-sync --no-clone-bundle --no-tags -j$(nproc --all). I get message error: RPC failed; curl 56 GnuTLS recv error (-24): Decryption has failed.
i can't finish compiling because java.lang.outofmemoryerror : heap size but i have already set the heap size to 8gb

Categories

Resources