7cc5c4f330d47060/tiktok-sparo
1
0
Fork 0
mirror of https://github.com/tiktok/sparo.git synced 2024-11-14 19:35:12 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

Tune up the website content

Browse source
This commit is contained in:
Pete Gonzalez 2024-03-04 23:51:56 -08:00
parent 12ddfb4710
commit a3f2ea9881
6 changed files with 108 additions and 58 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

32
apps/sparo/README.md
View file

@ -12,34 +12,18 @@
By default `git clone` will download every file in your Git repository, as well as the complete history of every file. For small repositories, that's no big deal. But as your monorepo accumulates projects and years of history, Git operations become slower and slower, until one day `git status` is taking 10 seconds or more. What to do?
Git provides these basic solutions that are easy to use in a medium sized repository:
- **Shallow clone** allows cloning only a few commits, but is generally only suitable for throwaway clones such as a CI job.
- **Partial clone** allows cloning without file contents (**blobless** clone) or even commit details (**treeless** clone), greatly accelerating your `git clone` time and allowing such details to be fetched during `git checkout`.
- **Large file storage (LFS)** can move binary files to a separate server, downloading them on demand during checkout. Configuration of LFS is tricky however and if done incorrectly may cause worse performance.
However, achieving good performance in a large repository requires more complex Git features such as:
- Git **filesystem monitor** and **background maintenance** are background processes that watch for changes and periodically prefetch server data. The user must manually register/unregister working directories and remember to "pause" the service when not needed.
- **Git worktrees** allow multiple working directories on your computer to share a single `.git` folder, avoiding the cost of multiple clones. However this feature comes with awkward limitations, for example the same branch can't be checked out in two worktrees, and Git hooks are also shared.
- **Sparse checkout** allows `git checkout` to extract a subset of files instead of the entire directory structure. Combined with partial clone, sparse checkout is the "battle axe" of Git optimization: although irrelevant projects and history will accumulate, your wait time will be proportional to the files you actually need.
## How does Sparo help?
<!-- Text below this line should stay in sync with the website index.md -->
<!-- ------------------------------------------------------------------ -->
**Sparo** optimizes performance of Git operations for your large frontend monorepo.
## Clone faster!
Sparo optimizes performance of Git operations for your large frontend monorepo.
## Key features
- **Familiar interface:** The `sparo` command-line interface (CLI) wrapper offers **better defaults** and **performance suggestions** without altering the familiar `git` syntax. (The native `git` CLI is also supported.)
- **A proven solution:** Git provides [quite a lot of ingredients](./pages/guide/git_features.md) for optimizing very large repos; Sparo is your recipe for combining these features intelligently.
- **Simplified sparse checkout:** Work with sparse checkout [profiles](./pages/guide/sparo_profiles.md) instead of confusing "cones" and globs
- **A proven solution:** Git provides [quite a lot of ingredients](https://tiktok.github.io/sparo/pages/reference/git_optimization/) for optimizing very large repos; Sparo is your recipe for combining these features intelligently.
- **Simplified sparse checkout:** Work with sparse checkout [profiles](https://tiktok.github.io/sparo/pages/guide/sparo_profiles/) instead of confusing "cones" and globs
- **Frontend integration:** Sparo leverages [Rush](https://rushjs.io/) and [PNPM](https://pnpm.io/) workspace configurations, including the ability to automatically checkout project dependencies
- **Dual workflows:** The `sparo-ci` tool implements a specialized checkout model optimized for continuous integration (CI) pipelines
- **Extra safeguards**: Avoid common Git mistakes such as checkouts with staged files outside the active view
@ -58,3 +42,9 @@ npm install -g sparo
```
<!-- more to come later -->
## Links
- [CHANGELOG.md](
https://github.com/tiktok/sparo/blob/main/apps/sparo/CHANGELOG.md) - Find
out what's new in the latest version

43
apps/website/docs/index.md
View file

@ -28,8 +28,8 @@ Sparo optimizes performance of Git operations for your large frontend monorepo.
## Key features
- **Familiar interface:** The `sparo` command-line interface (CLI) wrapper offers **better defaults** and **performance suggestions** without altering the familiar `git` syntax. (The native `git` CLI is also supported.)
- **A proven solution:** Git provides [quite a lot of ingredients](./pages/reference/git_optimization.md) for optimizing very large repos; Sparo is your recipe for combining these features intelligently.
- **Simplified sparse checkout:** Work with sparse checkout [profiles](./pages/guide/sparo_profiles.md) instead of confusing "cones" and globs
- **A proven solution:** Git provides [quite a lot of ingredients](https://tiktok.github.io/sparo/pages/reference/git_optimization/) for optimizing very large repos; Sparo is your recipe for combining these features intelligently.
- **Simplified sparse checkout:** Work with sparse checkout [profiles](https://tiktok.github.io/sparo/pages/guide/sparo_profiles/) instead of confusing "cones" and globs
- **Frontend integration:** Sparo leverages [Rush](https://rushjs.io/) and [PNPM](https://pnpm.io/) workspace configurations, including the ability to automatically checkout project dependencies
- **Dual workflows:** The `sparo-ci` tool implements a specialized checkout model optimized for continuous integration (CI) pipelines
- **Extra safeguards**: Avoid common Git mistakes such as checkouts with staged files outside the active view
@ -44,42 +44,60 @@ Sparo optimizes performance of Git operations for your large frontend monorepo.
Try out Sparo in 5 easy steps:
1. Ensure you are using the latest Git version. For macOS, we recommend to use [brew install git](https://git-scm.com/download/mac). For other operating systems, see the [Git documentation](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) for instructions.
1. _**Upgrade to the latest Git version!**_ For macOS, we recommend to use [brew install git](https://git-scm.com/download/mac). For other operating systems, see the [Git documentation](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) for instructions.
2. Clone your [RushJS](https://rushjs.io/) monorepo:
2. For this demo, we'll use the Azure SDK which is a large public [RushJS](https://rushjs.io/) monorepo from GitHub. The following command will check out the [skeleton folders](./pages/reference/skeleton_folders.md) but not the source code:
```shell
sparo clone https://github.com/my-company/my-monorepo.git
```
sparo clone https://github.com/Azure/azure-sdk-for-js.git
👉 _For a real world demo, try this repo:_
[https://github.com/Azure/azure-sdk-for-js.git](https://github.com/Azure/azure-sdk-for-js.git)
cd azure-sdk-for-js
```
> 💡 Support for PNPM and Yarn workspaces is planned but not implemented yet. Contributions welcome!
3. Define a [Sparo profile](./pages/configs/profile_json.md) describing the subset of repository folders for Git sparse checkout. Here is a basic example:
3. Define a [Sparo profile](./pages/configs/profile_json.md) describing the subset of repository folders for Git sparse checkout.
```shell
# Writes a template to common/sparo-profiles/my-team.json
sparo init-profile --profile my-team
```
Edit the created **my-team.json** file to add this selector:
**common/sparo-profiles/my-team.json**
```json
{
"selections": [
{
// This demo profile will check out the "@azure/arm-commerce" project
// and all of its dependencies:
"selector": "--to",
"argument": "my-rush-project"
"argument": "@azure/arm-commerce"
}
]
}
```
The `--to` [project selector](https://rushjs.io/pages/developer/selecting_subsets/#--to) instructs Sparo to checkout all dependencies in the workspace that are required to build `my-rush-project`.
👉 _If you're demoing **azure-sdk-for-js**, replace `my-rush-project` with `@azure/arm-commerce`._
4. Check out your Sparo profile:
4. After saving your changes to **my-team.json**, now it's time to apply it:
```shell
sparo checkout --profile my-team
```
Try it out! For example:
```shell
rush install
# The build should succeed because Sparo ensured that dependency projects
# were included in the sparse checkout:
rush build --to @azure/arm-commerce
```
5. For everyday work, consider choosing [mirrored subcommands](./pages/commands/overview.md) such as `sparo revert` instead of `git revert`. The Sparo wrapper provides (1) better defaults, (2) suggestions for better performance, and (3) optional anonymized performance metrics.
Examples:
@ -90,3 +108,4 @@ Try out Sparo in 5 easy steps:
sparo commit -m "Example command"
```
👍👍 This concludes the **Quick Demo.** For a more detailed walkthrough, proceed to [Getting Started](./pages/guide/getting_started.md).

63
apps/website/docs/pages/guide/getting_started.md
View file

@ -1,41 +1,53 @@
---
title: Getting Started
title: Getting started
---
## Everyday workflow
In this tutorial we'll revisit the [Quick Demo](../../index.md#quick-demo) steps, but this time examining the Sparo workflow in more detail.
### Step 1: Upgrade Git
## Step 1: Upgrade Git
Many Git optimizations are relatively new and not available in older versions of the software. For macOS, we recommend to use [brew install git](https://git-scm.com/download/mac). For other operating systems, see the [Git documentation](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) for instructions.
Remember to upgrade to the latest Git version! Many Git optimizations are relatively new and not available in older versions of the software.
### Step 2: Clone your Rush monorepo
For macOS, we recommend to use [brew install git](https://git-scm.com/download/mac). For other operating systems, see the [Git documentation](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) for instructions.
## Step 2: Clone your Rush monorepo
Clone your [RushJS](https://rushjs.io/) monorepo:
```shell
sparo clone https://github.com/my-company/my-monorepo.git
cd my-monorepo
```
👉 _For a real world demo, try this repo:_
👉 _For a real world demo, try cloning this repo:_
[https://github.com/Azure/azure-sdk-for-js.git](https://github.com/Azure/azure-sdk-for-js.git)
> 💡 Support for PNPM and Yarn workspaces is planned but not implemented yet. Contributions welcome!
**Behind the scenes:**
**How "sparo clone" optimizes:**
- Only the default branch (usually `main`) is fetched.
- Only the default branch is fetched (typically the `main` branch). This significantly reduces the download size.
- Git blobless [partial clone](../reference/git_optimization.md) is enabled to postpone downloading file contents.
- Git [sparse checkout](../reference/git_optimization.md) is used to clone only the ["skeleton" folders](../reference/skeleton_folders.md), which includes all workspace **package.json** files, but excludes the source code subfolders.
- Git [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) is used to clone only the ["skeleton" folders](../reference/skeleton_folders.md), which includes all workspace **package.json** files, but excludes the source code subfolders.
- Sparse checkout is configured for the more efficient "cone mode".
- Sparse checkout is configured for the more efficient ["cone mode"](https://git-scm.com/docs/git-sparse-checkout#_internalsnon_cone_problems).
- To understand exactly what actions and Git operations are being performed, invoke `sparo --debug clone` instead of `sparo clone`.
**Tip:** To inspect what actions and Git operations are being performed, invoke `sparo --debug clone` instead of `sparo clone`.
### Step 3: Create a sparse profile
> 💡 Support for PNPM and Yarn workspaces is planned but not implemented yet. Contributions welcome!
Define a [Sparo profile](../configs/profile_json.md) describing the subset of repository folders for Git sparse checkout. Here is a basic example:
## Step 3: Create a sparse profile
Define a [Sparo profile](../configs/profile_json.md) describing the subset of repository folders for Git sparse checkout.
```shell
# Writes a template to common/sparo-profiles/my-team.json
sparo init-profile --profile my-team
```
Edit the created **my-team.json** file to add a selector. For example:
**common/sparo-profiles/my-team.json**
```json
@ -48,22 +60,31 @@ Define a [Sparo profile](../configs/profile_json.md) describing the subset of re
]
}
```
The `--to` [project selector](https://rushjs.io/pages/developer/selecting_subsets/#--to) instructs Sparo to checkout all dependencies in the workspace that are required to build `my-rush-project`.
👉 _If you're demoing **azure-sdk-for-js**, replace `my-rush-project` with `@azure/arm-commerce`._
### Step 4: Check out your Sparo profile
In the above example, the `--to` [project selector](https://rushjs.io/pages/developer/selecting_subsets/#--to) instructs Sparo to checkout all dependencies in the workspace that are required to build `my-rush-project`.
The `--profile` parameter can be included with `sparo checkout` (and in the future also `sparo clone` and `sparo pull`). This parameter specifies the name of the JSON file to be selected. You can also combine multiple profiles (`sparo checkout --profile p1 --profile p2`), in which case the union of their selections will be used. (Combining profiles is an advanced scenario, but useful for example if your pull request will impact sets of projects belonging to multiple teams.)
```shell
# Commit your profile to Git. (This step was skipped in the Quick Demo.)
# Sparo profiles should generally be stored in Git, since this enables
# you to move between branches without worrying about which projects
# exist in a given branch.
sparo add .
sparo commit -m "Created a new Sparo profile"
```
## Step 4: Check out your Sparo profile
The `--profile` parameter can be included with `sparo checkout` (and in the future also `sparo clone` and `sparo pull`). This parameter specifies the name of the JSON file to be selected. You can also combine multiple profiles (`sparo checkout --profile p1 --profile p2`), in which case the union of their selections will be used. Combining profiles is an advanced scenario, but useful for example if your pull request will impact sets of projects belonging to multiple teams.
**Sparse checkout based on common/sparo-profiles/my-team.json**
```shell
sparo checkout --profile my-team
```
**Behind the scenes:**
**More about "sparo checkout":**
- Sparo automatically generates the `$GIT_DIR/info/sparse-checkout` configuration automatically based on your profile selections. To avoid conflicts, while using Sparo do not edit this file directly or rewrite it using other tools such as `git sparse-checkout`.
- Sparo automatically generates Git's `$GIT_DIR/info/sparse-checkout` [config file](https://git-scm.com/docs/git-sparse-checkout#_internalssparse_checkout) based on your profile selections. To avoid conflicts, do not edit this file directly or rewrite it using other tools such as `git sparse-checkout`. (Doing so won't break anything, but it may interfere with Sparo operations.)
- To checkout just the skeleton (returning to the initial state from Step 1 where no profile is chosen yet), specify `--no-profile` instead of `--profile NAME`.
@ -73,7 +94,7 @@ sparo checkout --profile my-team
sparo checkout --add-profile p2
```
### Step 5: Use the mirrored subcommands
## Step 5: Use the mirrored subcommands
For everyday work, consider choosing [mirrored subcommands](../commands/overview.md) such as `sparo revert` instead of `git revert`. The Sparo wrapper provides (1) better defaults, (2) suggestions for better performance, and (3) optional anonymized performance metrics.

16
apps/website/docs/pages/guide/sparo_profiles.md
View file

@ -2,7 +2,9 @@
title: Sparo profiles
---
Git's sparse checkout feature normally relies on a collection of glob patterns that are stored in the `.git/info/sparse-checkout` config file. Normal glob syntax proved to be too inefficient, so Git instead uses a ["cone mode"](https://git-scm.com/docs/git-sparse-checkout#_internalsnon_cone_problems) glob interpretation that ignores file-matching patterns and only matches directories.
## Background
Git's sparse checkout feature normally relies on a collection of glob patterns that are stored in the `.git/info/sparse-checkout` config file. The Git maintainers found that regular glob syntax was too inefficient, so they introduced a ["cone mode"](https://git-scm.com/docs/git-sparse-checkout#_internalsnon_cone_problems) glob interpretation that ignores file-matching patterns and only matches directories.
The syntax looks something like this:
@ -17,7 +19,9 @@ The syntax looks something like this:
/apps/my-app/_/
```
To simplify management, the `git sparse-checkout` command line provides convenient ways to add/remove patterns from this file. However, in a large monorepo with hundreds of projects, managing these globs would nonetheless be confusing and error-prone.
To simplify management, Git also provides a `git sparse-checkout` command that simplifies the syntax for adding/removing patterns from this file. However, in a large monorepo with hundreds of projects, managing these globs would nonetheless be confusing and error-prone.
## Sparo improves sparse checkout
Sparo makes life easier by generating the `.git/info/sparse-checkout` configuration automatically from config files called **profiles.** This offers many benefits:
@ -82,7 +86,7 @@ sparo checkout --add-profile team-b
sparo checkout --add-profile team-c
```
How to checkout no profile at all? That is, how to return to the initial state of a clean `sparo clone` that only includes the [skeleton](../reference/skeleton_folders.md) folders? If `--profile` is entirely omitted, then `sparo checkout` will preserve the existing profile selection. Instead, the `--no-profile` parameter is needed:
How to checkout no profile at all? That is, how to return to the initial state of a clean `sparo clone` that only includes the [skeleton](../reference/skeleton_folders.md) folders? The answer is to use the `--no-profile` parameter:
```shell
# NOT IMPLEMENTED YET - check out just the skeleton folders
@ -90,6 +94,8 @@ How to checkout no profile at all? That is, how to return to the initial state o
sparo checkout --no-profile
```
If `sparo checkout` without `--profile` or `--add-profile` or `--no-profile`, then the existing profile selection is preserved. In other words, your profile choices are generally "sticky" across commands.
## Querying profiles
@ -105,3 +111,7 @@ sparo list-profiles --project example-app
# (combining it with the existing profile).
sparo checkout --add-profile example-profile
```
## See also
- [&lt;profile-name&gt;.json](../configs/profile_json.md) config file

7
apps/website/docs/pages/support/help.md
View file

@ -5,3 +5,10 @@ title: Getting help
Please [create a GitHub issue](https://github.com/tiktok/sparo/issues/new/choose) to report any problems or feature requests.
For general questions, please use our [GitHub Discussions](https://github.com/tiktok/sparo/discussions) forum.
## Troubleshooting tips
- UPGRADE TO THE LATEST GIT. Sparo normally reports an error if your Git version is too old. If the minimum Git version is not high enough, let us know!
- If an operation is failing, use `--debug` to investigate. For example, `sparo --debug clone http://my-repo` instead of `sparo clone http://my-repo`.

5
apps/website/docs/pages/support/news.md
View file

@ -2,4 +2,7 @@
title: What's new
---
To find out what's changed in the latest release, please consult the [CHANGELOG.md](https://github.com/tiktok/sparo/blob/main/apps/sparo/CHANGELOG.md) notes.
To find out what's changed in the latest release, please consult the change log:
[CHANGELOG.md](https://github.com/tiktok/sparo/blob/main/apps/sparo/CHANGELOG.md)