Internal Server Error 😢 🔥 #325
Comments
nelsonic
added
help wanted
|
Yep. dwyl/mvp#440 |
|
Suspect it has something to do with this email received from |
|
Also confirm this error on my end 😢 |
|
Lame. Thanks |
|
Thankfully we have a "Staging" instance that still works flawlessly: https://authdemo.fly.dev/ 
Just re-deploying now because the machine restart didn't work to update the environment variable. 🙃 |
|
MVP now redirects to authdemo: Works for me: 
@iteles / @LuchoTurtle please confirm you're able to access your |
|
Definitely still going to try and recover the |
|
Interesting. It's working for me on Firefox desktop! On Chrome desktop, I'm still getting an internal server error straight away (on the start page, not even getting to the login portion) even after clearing cookies and cached data 🤷♀️ But fixed for now, no point troubleshooting this further until we're back on |
|
@iteles In Chrome, if you visit https://mvp.fly.dev/ what do you see? |
|
|
Well that's mega frustrating! Thank you |
|
Works on my phone:
but doesn’t look great. Some tidy up to be done. ⏳ |
|
Works on Chrome on my phone but I can't see any of my items. |
|
Ok. So that means your |
|
Only works if I clear my cookies. I'm doing this on the web. |
|
Trying to re-deploy n@MBP auth % fly deploy --verbose
==> Verifying app config
Validating /auth/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Waiting for remote builder fly-builder-solitary-rain-5830... 🌎 |
|
Useless! 😢 WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-solitary-rain-5830`
WARN Failed to start remote builder heartbeat: remote builder app unavailable
WARN failed to create build in graphql: returned error 504 Gateway Timeout: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
Error: failed to fetch an image or build from source: error connecting to docker: server returned a non-200 status code: 503 |
|
Just in case I was going crazy, I tried to deploy the fly deploy --verboseOutput: ==> Verifying app config
Validating /auth/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-solitary-rain-5830 ready
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 1.8s (30/30) FINISHED
=> [internal] load build definition from Dockerfile 1.0s
=> => transferring dockerfile: 32B 1.0s
=> [internal] load .dockerignore 1.0s
=> => transferring context: 34B 1.0s
=> [internal] load metadata for docker.io/library/debian:bullseye-20210902-slim 0.3s
=> [internal] load metadata for docker.io/hexpm/elixir:1.12.3-erlang-24.1.5-debian-bullseye-20210902-slim 0.4s
=> [internal] load build context 0.3s
=> => transferring context: 29.65kB 0.3s
=> [stage-1 1/6] FROM docker.io/library/debian:bullseye-20210902-slim@sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d79204ace20455a6 0.0s
=> [builder 1/18] FROM docker.io/hexpm/elixir:1.12.3-erlang-24.1.5-debian-bullseye-20210902-slim@sha256:a2447f82e9125abb7edb0708eb51da638d8f40e7fda4b3ed5151bee883d476be 0.0s
=> CACHED [stage-1 2/6] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales && apt-get clean && rm -f /var/lib/apt/lists/*_* 0.0s
=> CACHED [stage-1 3/6] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen 0.0s
=> CACHED [stage-1 4/6] WORKDIR /app 0.0s
=> CACHED [stage-1 5/6] RUN chown nobody /app 0.0s
=> CACHED [builder 2/18] RUN apt-get update -y && apt-get install -y build-essential git && apt-get clean && rm -f /var/lib/apt/lists/*_* 0.0s
=> CACHED [builder 3/18] WORKDIR /app 0.0s
=> CACHED [builder 4/18] RUN mix local.hex --force && mix local.rebar --force 0.0s
=> CACHED [builder 5/18] COPY .env_sample ./ 0.0s
=> CACHED [builder 6/18] COPY mix.exs mix.lock ./ 0.0s
=> CACHED [builder 7/18] RUN mix deps.get --only prod 0.0s
=> CACHED [builder 8/18] RUN mkdir config 0.0s
=> CACHED [builder 9/18] COPY config/config.exs config/prod.exs config/ 0.0s
=> CACHED [builder 10/18] RUN mix deps.compile 0.0s
=> CACHED [builder 11/18] COPY priv priv 0.0s
=> CACHED [builder 12/18] COPY assets assets 0.0s
=> CACHED [builder 13/18] RUN mix assets.deploy 0.0s
=> CACHED [builder 14/18] COPY lib lib 0.0s
=> CACHED [builder 15/18] RUN mix compile 0.0s
=> CACHED [builder 16/18] COPY config/runtime.exs config/ 0.0s
=> CACHED [builder 17/18] COPY rel rel 0.0s
=> CACHED [builder 18/18] RUN mix release 0.0s
=> CACHED [stage-1 6/6] COPY --from=builder --chown=nobody:root /app/_build/prod/rel/auth ./ 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:5ba72f99464041b84ed2e1491eb709c1518dff88852706229e8d82f0f3af2d4c 0.0s
=> => naming to registry.fly.io/authprod:deployment-01HEADJTX39GT10NA7TY49HG02 0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/authprod]
7b5d131176f6: Layer already exists
4d5c3ee4879b: Layer already exists
0166b8c07570: Layer already exists
4dfd7d2f8a33: Layer already exists
11a55e4db61e: Layer already exists
ba5a5fe43301: Layer already exists
deployment-01HEADJTX39GT10NA7TY49HG02: digest: sha256:f5d586ae7234d29f8ef485ce6c06e5ec469f6ae66d77886459780fc172337147 size: 1575
--> Pushing image done
image: registry.fly.io/authprod:deployment-01HEADJTX39GT10NA7TY49HG02
image size: 120 MB
Watch your deployment at https://fly.io/apps/authprod/monitoring
Running authprod release_command: /app/bin/migrate
-------
✖ release_command failed
-------
Error release_command failed running on machine 48ed037a343368 with exit code 1.
Check its logs: here's the last 100 lines below, or run 'fly logs -i 48ed037a343368':
Pulling container image registry.fly.io/authprod:deployment-01HEADJTX39GT10NA7TY49HG02
Successfully prepared image registry.fly.io/authprod:deployment-01HEADJTX39GT10NA7TY49HG02 (5.834199334s)
[ 0.041401] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
[ 0.042773] PCI: Fatal: No config space access function found
INFO Starting init (commit: 15238e9)...
INFO Preparing to run: `/app/bin/migrate` as nobody
INFO [fly api proxy] listening at /.fly/api
2023/11/03 11:02:57 listening on [fdaa:0:42c6:a7b:1be:99c2:78d9:2]:22 (DNS: [fdaa::3]:53)
11:02:58.710 [error] Postgrex.Protocol (#PID<0.136.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (top2.nearest.of.authprodb.internal:5432): non-existing domain - :nxdomain
11:02:58.710 [error] Postgrex.Protocol (#PID<0.135.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (top2.nearest.of.authprodb.internal:5432): non-existing domain - :nxdomain
11:02:59.943 [error] Postgrex.Protocol (#PID<0.136.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (top2.nearest.of.authprodb.internal:5432): non-existing domain - :nxdomain
11:03:00.660 [error] Could not create schema migrations table. This error usually happens due to the following:
* The database does not exist
* The "schema_migrations" table, which Ecto uses for managing
migrations, was defined by another library
* There is a deadlock while migrating (such as using concurrent
indexes with a migration_lock)
To fix the first issue, run "mix ecto.create".
To address the second, you can run "mix ecto.drop" followed by
"mix ecto.create". Alternatively you may configure Ecto to use
another table and/or repository for managing migrations:
config :auth, Auth.Repo,
migration_source: "some_other_table_for_schema_migrations",
migration_repo: AnotherRepoForSchemaMigrations
The full error report is shown below.
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2975ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
1. Ensuring your database is available and that you can connect to it
2. Tracking down slow queries and making sure they are running fast enough
3. Increasing the pool_size (although this increases resource consumption)
4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
See DBConnection.start_link/2 for more information
(ecto_sql 3.7.2) lib/ecto/adapters/sql.ex:760: Ecto.Adapters.SQL.raise_sql_call_error/1
(elixir 1.12.3) lib/enum.ex:1582: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql 3.7.2) lib/ecto/adapters/sql.ex:852: Ecto.Adapters.SQL.execute_ddl/4
(ecto_sql 3.7.2) lib/ecto/migrator.ex:678: Ecto.Migrator.verbose_schema_migration/3
(ecto_sql 3.7.2) lib/ecto/migrator.ex:504: Ecto.Migrator.lock_for_migrations/4
(ecto_sql 3.7.2) lib/ecto/migrator.ex:419: Ecto.Migrator.run/4
(ecto_sql 3.7.2) lib/ecto/migrator.ex:146: Ecto.Migrator.with_repo/3
(auth 1.6.6) lib/auth/release.ex:12: anonymous fn/2 in Auth.Release.migrate/0
INFO Main child exited normally with code: 1
WARN Reaped child process with pid: 359 and signal: SIGUSR1, core dumped? false
INFO Starting clean up.
WARN hallpass exited, pid: 306, status: signal: 15 (SIGTERM)
2023/11/03 11:03:01 listening on [fdaa:0:42c6:a7b:1be:99c2:78d9:2]:22 (DNS: [fdaa::3]:53)
[ 5.255192] reboot: Restarting system
machine restart policy set to 'no', not restarting
-------
Error: release command failed - aborting deployment. error release_command machine 48ed037a343368 exited with non-zero status of 1
|
|
Posted Forum topic: https://community.fly.io/t/where-is-our-data-postgres-database-volume-0mb/16247 Snapshot: Full text of my forum topic in case it gets deleted: Opening this question in case anyone `else` is in the same boat (having their `Nomad`-based `Postgres` instance killed ...)
# Context
We deployed our [`auth`](https://github.com/dwyl/auth) (`Elixir/Phoenix`) app to Fly.io in 2022. :rocket:
It was a good experience and worked flawlessly until they killed it. :neutral_face:
# Fly.io App Disabled 💀
We received the following email from Fly.io:
## `Fulltext` of the email for Google indexing/findability 📧
```md
Hi. We’re in the process of removing Nomad from Fly.io. This means any apps that haven’t been upgraded to Apps V2 will no longer function.
Your app authprodb is one of those apps. Here’s what this means for you:
1. authprodb is currently disabled.
- That means the Nomad instances (virtual machines)
that ran your app have been shut down, but that the app’s configuration still exists.
2. Your app is technically now on the “machines” platform, rather than “nomad.”
- That means that any subsequent fly deploy on your app will attempt to launch Fly Machines.
- In many cases, a deploy will immediately work. In some cases, there may be some additional configuration required to get your app running on Machines.
The community forum is a great place to get help troubleshooting your deployment if you get stuck.
3. Nothing has been deleted.
- Your app’s Docker image still exists, and any Volumes that were attached to your app still exist.
Thanks for bearing with us throughout this whole migration saga. It’s not been easy, but the results should soon speak for themselves.
If you have any questions or comments, we’ve posted a similar announcement on the community forum.
Thanks, Your Friends at Fly.ioThey key point of the email: "Nothing has been deleted" ... 💭So I dived into trying to recover the database. fly deploy --verboseIt errors because the Error: release command failed - aborting deployment. error release_command machine 48ed037a343368 exited with non-zero status of 1Full log output in: #325 (comment) After a couple of hours searching and trying everything I could think of with zero progress. Question: Why is the Volume Used
|
|
Replied to the support email:
Ball is in their court. 🎾 AFK to help parents with something for the next couple of hours. 🤞 |
|
Meanwhile I've wasted my whole morning on this ... ⏳ 🔥 😢 |
|
Followed-up with |
|
Got a couple of replies on the forum: Tried creating a fork from the second volume: fly postgres create --fork-from authprodb:vol_1g67340ggznrydxwOutput: Connect: fly postgres connect -a authprodb3
I replied on the thread. |
|
Got a reply on the forum (but still nothing in email ...) Only saw it now ... didn't get a notification from the forum. We need a "Reminders" feature. Going to try the command now: fly postgres create --name new-db --vm-size shared-cpu-2x --volume-size 10 --initial-cluster-size 1 --region CHOOSE --org your-org --image-ref registry-1.docker.io/flyio/postgres:13 --stolon --snapshot-id vs_yaddayaddayaddaRe-ran the command to check the Using the latest snapshot: Creating app...
Setting secrets on app new-db...
Restoring 1 of 1 machines with image registry-1.docker.io/flyio/postgres:13
Waiting for machine to start...Gotta go be a Dad for a few hours. BRB. ⏳ |
great. going to have to run the command again ... |
|
Posted a comment on the forum topic to this effect: Not much |
|
Given the error above, I had deleted But sadly, Fly.io is not fully operational right now: https://fly.io/dashboard/dwyl-auth/status 
So the response is: Will need to revisit this in the morning. ⏳ |
|
At home now and still getting error trying to revive the DB: Creating postgres cluster in organization dwyl-auth-546
Creating app...
Setting secrets on app new-db...
Restoring 1 of 1 machines with image registry-1.docker.io/flyio/postgres:13
Waiting for machine to start...
Machine 784e679b453408 is created
==> Monitoring health checks
Waiting for 784e679b453408 to become healthy (started, 0/3)
Error: context deadline exceeded |
|
But when visiting the dashboard, we see that the Replied on the thread: https://community.fly.io/t/where-is-our-data-postgres-database-volume-0mb/16247/16?u=dwyl_auth |
|
On the advice of the Daniel https://community.fly.io/t/where-is-our-data-postgres-database-volume-0mb/16247/18 I created a new DB instance to attempt to recover the data ... https://fly.io/dashboard/dwyl-auth-546 It didn't work and the thread was automatically closed without any resolution. 😢 
The
|
|
Going to attempt to host the Meanwhile our Auth App has a mountain of credits: https://fly.io/dashboard/dwyl-auth-546 But just to say I'm tackling this now! ⏳ (or at least investing an hour into this and seeing how far I get ...) |
All running. 🆗 |
|
Going to detach the (empty) DB: https://fly.io/docs/postgres/managing/attach-detach/#detach-a-fly-app Example: fly postgres detach <postgres app name> --app <app name>Ours: fly postgres detach authprodb --app authprodGot: Error: no 6pn ips founds for authprodb appDidin't expect it to be easy ... 🙄 |
|
Exact error message for forum title/topic: Reading now ... ⏳ |
|
Turns out the fly postgres users list --app authprodb
Error: no active leader foundFly Postgres is Total Garbage. 🗑️ 😢 Why aren't these status checks all automatic?! 🤦♂️ |
|
Interrupted by children. 🧒 ❤️ So no further progress will be made on this today. |
nelsonic
moved this from ⏳Awaiting Review
to 🔖 Ready for Development
in dwyl app kanban
==> ecto
Compiling 56 files (.ex)
warning: missing parentheses for expression following "do:" keyword.
Parentheses are required to solve ambiguity inside keywords.
This error happens when you have function calls without parentheses inside keywords. For example:
function(arg, one: nested_call a, b, c)
function(arg, one: if expr, do: :this, else: :that)
In the examples above, we don't know if the arguments "b" and "c" apply to the function "function" or "nested_call".
Or if the keywords "do" and "else" apply to the function "function" or "if".
You can solve this by explicitly adding parentheses:
function(arg, one: if(expr, do: :this, else: :that))
function(arg, one: nested_call(a, b, c))
Ambiguity found at:
│
568 │ do: raise ArgumentError, "cast/3 expects a list of atom keys, got: `#{inspect key}`"
│ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
│
└─ lib/ecto/changeset.ex:568
warning: missing parentheses for expression following "do:" keyword.
Parentheses are required to solve ambiguity inside keywords.
This error happens when you have function calls without parentheses inside keywords. For example:
function(arg, one: nested_call a, b, c)
function(arg, one: if expr, do: :this, else: :that)
In the examples above, we don't know if the arguments "b" and "c" apply to the function "function" or "nested_call".
Or if the keywords "do" and "else" apply to the function "function" or "if".
You can solve this by explicitly adding parentheses:
function(arg, one: if(expr, do: :this, else: :that))
function(arg, one: nested_call(a, b, c))
Ambiguity found at:
│
445 │ do: raise ArgumentError,
│ ~~~~~~~~~~~~~~~~~~~~~~~~
│
└─ lib/ecto/association.ex:445
warning: missing parentheses for expression following "do:" keyword.
Parentheses are required to solve ambiguity inside keywords.
This error happens when you have function calls without parentheses inside keywords. For example:
function(arg, one: nested_call a, b, c)
function(arg, one: if expr, do: :this, else: :that)
In the examples above, we don't know if the arguments "b" and "c" apply to the function "function" or "nested_call".
Or if the keywords "do" and "else" apply to the function "function" or "if".
You can solve this by explicitly adding parentheses:
function(arg, one: if(expr, do: :this, else: :that))
function(arg, one: nested_call(a, b, c))
Ambiguity found at:
│
2904 │ raise ArgumentError, "cannot add constraint to changeset because it does not have a source, got: #{inspect data}"
│ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
│
└─ lib/ecto/changeset.ex:2904
warning: missing parentheses for expression following "do:" keyword. Parentheses are required to solve ambiguity inside keywords.
This error happens when you have function calls without parentheses inside keywords. For example:
function(arg, one: nested_call a, b, c)
function(arg, one: if expr, do: :this, else: :that)
In the examples above, we don't know if the arguments "b" and "c" apply to the function "function" or "nested_call".
Or if the keywords "do" and "else" apply to the function "function" or "if".
You can solve this by explicitly adding parentheses:
function(arg, one: if(expr, do: :this, else: :that))
function(arg, one: nested_call(a, b, c))
Ambiguity found at:
│
2906 │ raise ArgumentError, "cannot add constraint because a changeset was not supplied, got: #{inspect item}"
│ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
│
└─ lib/ecto/changeset.ex:2906
warning: atom ::: must be written between quotes, as in :"::", to avoid ambiguity
│
260 │ {:::, _, [left, _]} -> is_integer(left) or is_binary(left)
│ ~
│
└─ lib/ecto/query/builder.ex:260:8
warning: missing parentheses for expression following "do:" keyword.
Parentheses are required to solve ambiguity inside keywords.
This error happens when you have function calls without parentheses inside keywords. For example:
function(arg, one: nested_call a, b, c)
function(arg, one: if expr, do: :this, else: :that)
In the examples above, we don't know if the arguments "b" and "c" apply to the function "function" or "nested_call".
Or if the keywords "do" and "else" apply to the function "function" or "if".
You can solve this by explicitly adding parentheses:
function(arg, one: if(expr, do: :this, else: :that))
function(arg, one: nested_call(a, b, c))
Ambiguity found at:
│
621 │ do: raise ArgumentError, "a changeset with action #{inspect given} was given to #{inspect repo}.#{action}/2"
│ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
│
└─ lib/ecto/repo/schema.ex:621
== Compilation error in file lib/ecto/query.ex ==
** (Kernel.TypespecError) lib/ecto/query.ex:428: type dynamic/0 is a built-in type and it cannot be redefined
(elixir 1.17.2) lib/kernel/typespec.ex:969: Kernel.Typespec.compile_error/2
(stdlib 5.2.3) lists.erl:1599: :lists.foldl_1/3
(elixir 1.17.2) lib/kernel/typespec.ex:226: Kernel.Typespec.translate_typespecs_for_module/2
could not compile dependency :ecto, "mix compile" failed. Errors may have been logged above.
You can recompile this dependency with "mix deps.compile ecto --force",
update it with "mix deps.update ecto" or clean it with "mix deps.clean ecto"I obviously tried updating |
Is anyone else seeing an Internal Server Error on https://auth.dwyl.com/ ??
Todo
authprodapp. 🤞The text was updated successfully, but these errors were encountered: