The queries being done on the package search page are quite costly.
(Especially the default one ordered by "Popularity" when navigating to /packages)
Let's add the search results to the Redis cache:
Every result of a search query is being pushed to Redis until we hit our maximum of 50k.
An entry expires after 3 minutes before it's evicted from the cache.
Lifetime an Max values are configurable.
Signed-off-by: moson-mo <mo-son@mailbox.org>
We add a new config option for cookies with a 400 day lifetime.
AURLANG should survive longer for unauthenticated users.
Today they have to set this again after each browser restart.
(for users whose browsers wipe session cookies on close)
authenticated users don't need this cookie
since the setting is saved to the DB
Signed-off-by: moson-mo <mo-son@mailbox.org>
Data for packages that do not have a "Last Packager"
(e.g. because the user account was deleted)
should still be available from the /rpc and metadata archives.
Signed-off-by: moson-mo <mo-son@mailbox.org>
In my opinion, this kind of handling of transactions is pretty ugly.
The being said, we have issues with running into deadlocks on aur.al,
so this commit works against that immediate bug.
An ideal solution would be to deal with retrying transactions through
the `db.begin()` scope, so we wouldn't have to explicitly annotate
functions as "retry functions," which is what this commit does.
Closes#376
Signed-off-by: Kevin Morris <kevr@0cost.org>
As repeats of these traceback notifications were annoying some of
the devops staff, and it took coordination to share tracebacks with
developers, this commit removes that responsibility off of devops
by reporting tracebacks to Gitlab repositories in the form of issues.
- removed ServerErrorNotification
- removed notifications.postmaster configuration option
- added notifications.gitlab-instance option
- added notifications.error-project option
- added notifications.error-token option
- added aurweb.exceptions.handle_form_exceptions, a POST route decorator
Issues are filed confidentially. This change will need updates
in infrastructure's ansible configuration before this can be
applied to aur.archlinux.org.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change brings some new additions to our archives:
- SHA-256 .sha256 hexdigests
- We construct our archives in a tmpdir now and move them to the
archive destination when all are completed. This removes some
corrupted downloading when archiving is in-process.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, we restricted this to gunicorn to get it working on aur-dev.
This change makes it usable through any backend, and also no-op if
PROMETHEUS_MULTIPROC_DIR is not defined.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change removes cookie re-emission of AURLANG and AURTZ,
adds the AURREMEMBER cookie (the state of the "Remember Me"
checkbox on login), and re-emits AURSID based on the AURREMEMBER
cookie.
Previously, re-emission of AURSID was forcefully modifying
the expiration of the AURSID cookie. The introduction of
AURREMEMBER allows us to deduct the correct cookie expiration
timing based on configuration variables. With this addition,
we now re-emit the AURSID cookie with an updated expiration
based on the "Remember Me" checkbox on login.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, we were running a single ORM query for every single package
to check for its voted or notified states. Now, we perform a single
ORM query for each of the set of voted or notified packages in
relation with the request user.
This improves performance drastically at the expense of some
manual code additions and set-dependency; i.e. we add a bit
more complexity and roundabout way of getting our data.
Closes: https://gitlab.archlinux.org/archlinux/aurweb/-/issues/102
Signed-off-by: Kevin Morris <kevr@0cost.org>
For SQLAlchemy to automatically understand updates from the
external world, it must use an `autocommit=True` in its session.
This change breaks how we were using commit previously, as
`autocommit=True` causes SQLAlchemy to commit when a
SessionTransaction context hits __exit__.
So, a refactoring was required of our tests: All usage of
any `db.{create,delete}` must be called **within** a
SessionTransaction context, created via new `db.begin()`.
From this point forward, we're going to require:
```
with db.begin():
db.create(...)
db.delete(...)
db.session.delete(object)
```
With this, we now get external DB modifications automatically
without reloading or restarting the FastAPI server, which we
absolutely need for production.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This puts one more toward completion of the homepage
overall; we'll need to still implement the authenticated
user dashboard after this.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This function adds f"SameSite={value}" to each cookie's header
stored in response.
This is needed because starlette does not currently support
the `samesite` argument in Response.set_cookie. It is merged,
however, and waiting for next release.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We'll piggyback off of the current existing configuration item,
`disable_http_login`, to decide how we should submit cookies to
an HTTP response.
Previously, in `sso.py`, the http schema was used to make this
decision. There is an issue with that, however: We cannot actually
test properly if we depend on the https schema.
This change allows us to toggle `disable_http_login` to modify
the behavior of cookies sent with an http response to be secure.
We test this behavior in test/test_auth_routes.py#L81:
`test_secure_login(mock)`.
Signed-off-by: Kevin Morris <kevr@0cost.org>
AUR renders its own 404 Not Found page when a bad route
is encountered. Introducing the previous verification
caused an error in this case when setting a language
while viewing the Not Found page. So, instead of checking
through routes, just make sure that the next parameter
starts with a '/' character, which removes the possibility
of any cross attacks.
+ Removed aurweb.asgi.routes; no longer needed.
Signed-off-by: Kevin Morris <kevr@0cost.org>
+ Use User.LangPreference when there is no set AURSID
if request.user.is_authenticated is true.
+ Updated post /language to update LangPreference when
request.user.is_authenticated.
+ Restore language during test where we change it.
+ Added the user attribute to aurweb.testing.requests.Request.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This allows us to inspect things about the request we're rendering from.
* Use render_template(request, ...) in aurweb.routers.auth
Signed-off-by: Kevin Morris <kevr@0cost.org>
+ Mounted static files (at web/html) to /static.
+ Added AURWEB_VERSION to aurweb.config (this is used around HTML
to refer back to aurweb's release on git.archlinux.org), so we
need it easily accessible in the Python codebase.
+ Implemented basic Jinja2 partials to put together whole aurweb
pages. This may be missing some things currently and is a WIP
until this set is ready to be merged.
+ Added config [options] aurwebdir = YOUR_AUR_ROOT; this configuration
option should specify the root directory of the aurweb project.
It is used by various parts of the FastAPI codebase to target
project directories.
Added routes via aurweb.routers.html:
* POST /language: Set your session language.
* GET /favicon.ico: Redirect to /static/images/favicon.ico.
* Some browsers always look for $ROOT/favicon.ico to get an icon
for the page being loaded, regardless of a specified "shortcut
icon" given in a <link> directive.
* GET /: Home page; WIP.
* Updated aurweb.routers.html.language passes query parameters to
its next redirection.
When calling aurweb.templates.render_template, the context passed should
be formed via the aurweb.templates.make_context. See
aurweb.routers.html.index for an example of this.
Signed-off-by: Kevin Morris <kevr@0cost.org>
