Bump sqlalchemy to version 2.0.22
There are quite some changes that happened with v2.
We are currently relying on the "auto-commit" feature which was removed.
For the moment we can use a wrapper class to mimic the auto-commit
behavior allowing us to move to v2.
Ultimately, the (db) session management needs some overhaul though.
Signed-off-by: moson <moson@archlinux.org>
In my opinion, this kind of handling of transactions is pretty ugly.
The being said, we have issues with running into deadlocks on aur.al,
so this commit works against that immediate bug.
An ideal solution would be to deal with retrying transactions through
the `db.begin()` scope, so we wouldn't have to explicitly annotate
functions as "retry functions," which is what this commit does.
Closes#376
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change utilizes pytest-xdist to perform a multiproc test
run and reworks aurweb.db's code. We no longer use a global
engine, session or Session, but we now use a memo of engines
and sessions as they are requested, based on the PYTEST_CURRENT_TEST
environment variable, which is available during testing.
Additionally, this change strips several SQLite components
out of the Python code-base.
SQLite is still compatible with PHP and sharness tests, but
not with our FastAPI implementation.
More changes:
------------
- Remove use of aurweb.db.session global in other code.
- Use new aurweb.db.name() dynamic db name function in env.py.
- Added 'addopts' to pytest.ini which utilizes multiprocessing.
- Highly recommended to leave this be or modify `-n auto` to
`-n {cpu_threads}` where cpu_threads is at least 2.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Changes:
-------
- Add aurweb.db.get_session()
- Returns aurweb.db's global `session` instance
- Provides us a way to change the implementation of the session
instance without interrupting user code.
- Use aurweb.db.get_session() in session API methods
- Add docstrings to session API methods
- Refactor aurweb.db.delete
- Normalize aurweb.db.delete to an alias of session.delete
- Refresh instances in places we depend on their non-PK columns
being up to date.
Signed-off-by: Kevin Morris <kevr@0cost.org>
For SQLAlchemy to automatically understand updates from the
external world, it must use an `autocommit=True` in its session.
This change breaks how we were using commit previously, as
`autocommit=True` causes SQLAlchemy to commit when a
SessionTransaction context hits __exit__.
So, a refactoring was required of our tests: All usage of
any `db.{create,delete}` must be called **within** a
SessionTransaction context, created via new `db.begin()`.
From this point forward, we're going to require:
```
with db.begin():
db.create(...)
db.delete(...)
db.session.delete(object)
```
With this, we now get external DB modifications automatically
without reloading or restarting the FastAPI server, which we
absolutely need for production.
Signed-off-by: Kevin Morris <kevr@0cost.org>
With the addition of these two, some code has been swapped
to use these in some of the other db wrappers with an additional
autocommit kwarg in create and delete, to control batch
transactions.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Django uses a reference graph to determine the order
in table deletions that occur. Do the same here.
This commit also adds in the `REGEXP` sqlite function,
exactly how Django uses it in its reference graphing.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This rewrites the entire model base as declarative models.
This allows us to more easily customize overlay fields
in tables and is more common.
This effort also brought some DB violations to light which
this commit addresses.
Signed-off-by: Kevin Morris <kevr@0cost.org>
First off: This commit changes the default development database
backend to mysql. sqlite, however, is still completely supported
with the caveat that a user must now modify config.dev to use
the sqlite backend.
While looking into this, it was discovered that our SQLAlchemy
backend for mysql (mysql-connector) completely broke model
attributes when we switched to utf8mb4_bin (binary) -- it does
not correct the correct conversion to and from binary utf8mb4.
The new, replacement dependency mysqlclient does. mysqlclient
is also recommended in SQLAlchemy documentation as the "best"
one available.
The mysqlclient backend uses a different exception flow then
sqlite, and so tests expecting IntegrityError has to be modified
to expect OperationalError from sqlalchemy.exc.
So, for each model that we define, check keys that can't be
NULL and raise sqlalchemy.exc.IntegrityError if we have to.
This way we keep our exceptions uniform.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Additionally, add an optional **kwargs passing via make_relationship.
This allows us to use things like `uselist=False`, which was needed
for test/test_package.py.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Introduced `get|post` `/passreset` routes. These routes mimic the
behavior of the existing PHP implementation, with the exception of
HTTP status code returns.
Routes added:
GET /passreset
POST /passreset
Routers added:
aurweb.routers.accounts
* On an unknown user or mismatched resetkey (where resetkey must ==
user.resetkey), return HTTP status NOT_FOUND (404).
* On another error in the request, return HTTP status BAD_REQUEST (400).
Both `get|post` routes requires that the current user is **not**
authenticated, hence `@auth_required(False, redirect="/")`.
+ Added auth_required decorator to aurweb.auth.
+ Added some more utility to aurweb.models.user.User.
+ Added `partials/error.html` template.
+ Added `passreset.html` template.
+ Added aurweb.db.ConnectionExecutor functor for paramstyle logic.
Decoupling the executor logic from the database connection logic
is needed for us to easily use the same logic with a fastapi
database session, when we need to use aurweb.scripts modules.
At this point, notification configuration is now required to complete
tests involved with notifications properly, like passreset.
`conf/config.dev` has been modified to include [notifications] sendmail,
sender and reply-to overrides. Dockerfile and .gitlab-ci.yml have been
updated to setup /etc/hosts and start postfix before running tests.
* setup.cfg: ignore E741, C901 in aurweb.routers.accounts
These two warnings (shown in the commit) are not dangerous and a bi-product
of maintaining compatibility with our current code flow.
Signed-off-by: Kevin Morris <kevr@0cost.org>
+ Added aurweb.util module.
- Added make_random_string function.
+ Added aurweb.db.make_random_value function.
- Takes a model and a column and introspects them to figure out the
proper column length to create a random string for; then creates
a unique string for that column.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Takes sqlalchemy kwargs or stanzas:
query(Model, Model.Column == value)
query(Model, and_(Model.Column == value, Model.Column != "BAD!"))
Updated tests to reflect the new utility and a comment about upcoming
function deprecation is added to get_account_type().
From here on, phase out the use of get_account_type().
+ aurweb.db: Added create utility function
+ aurweb.db: Added delete utility function
The `delete` function can be used to delete a record by search
kwargs directly.
Example:
delete(User, User.ID == 6)
All three functions added in this commit are typically useful to
perform these operations without having to import aurweb.db.session.
Removes a bit of redundancy overall.
Signed-off-by: Kevin Morris <kevr@0cost.org>
+ Added Session class and global session object to aurweb.db,
these are sessions created by sqlalchemy ORM's sessionmaker
and will allow us to use declarative/imperative models.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The new schema was generated with sqlacodegen and then manually adjusted
to fit schema/aur-schema.sql faithfully, both in the organisation of the
code and in the SQL generated by SQLAlchemy.
Initializing the database now requires the new tool aurweb.initdb.
References to aur-schema.sql have been updated and the old schema
dropped.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
We support multiple database backends. Don't require Python modules for
all backends to be installed.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Instead of using relative imports, add support for installing the config
and db Python modules to a proper location using setuptools. Change all
git-interface scripts to access those modules from the search path.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
