Scale out error ingestion for EF (2/3): --error-ingestion-only host

[johnsimons]

Part 2 of 3 introducing scale-out of error ingestion for EF persistence. Depends on #<PR 1>.

Adds --error-ingestion-only, which runs a host that does nothing but drain the error
queue into the shared database. Several of these can run alongside one normal instance.
SQL Server and PostgreSQL only; the command refuses to start on any other storage.

Why this works at all

Ingestion never uses the NServiceBus endpoint. ErrorIngestion builds its own
TransportInfrastructure and receiver through the low-level transport API, which is why
competing consumers come for free, and after part 1 it forwards through that same
infrastructure's dispatcher. Everything else the ingestion path does after the receive is a
database write: event log, integration dispatch requests, retry claim cleanup, known
endpoints, failed import records.

The batch writer was already built for concurrent writers (INSERT ... ON CONFLICT /
MERGE WITH (HOLDLOCK), newer-wins guards, ordering by UniqueMessageId for consistent
lock order), and ErrorIngestionConcurrencyTests already covers it.

No NServiceBus endpoint

The endpoint is not hosted. Its only remaining role would have been as a registration
source for other code in the process, so the host supplies the two things that actually
needed it: HostInformation (derived from the machine name, so each node reports its own
custom check row) and CriticalError (logs and stops the application).

A send-only endpoint looks like the safer option but is worse. SendOnly() drops
ReceiveAddresses anyway, which the EF SubscriptionStorage registered in
BasePersistence needs, and with the endpoint running the subscription feature activates
on any transport without native pub/sub (SQL Server, PostgreSQL, ASQ, MSMQ) and resolves a
store it cannot construct. Dropping the endpoint avoids that entirely.

What is switched off, and why

Component	Reason
EventDispatcherHostedService	The drain selects, publishes, then deletes with no row claim, so a second dispatcher publishes every integration event twice. Correctness, not tidiness.
ReturnToSenderDequeuer	A second staging queue receiver would steal messages from a retry batch. Gated on the mode flag rather than RunRetryProcessor, because it is the only writer of ErrorQueueNameCache.ResolvedErrorAddress, which EditHandler reads.
RetentionSweeper	Only one host should sweep. New PersistenceSettings.RunRetentionSweep.
HeartbeatEndpointSettingsSyncHostedService, heartbeat checking	A node that receives no heartbeats would report every endpoint dead. The monitor is still warmed from persistence, because the error enricher asks it whether an endpoint is new.
AddEmailNotifications, AddLicenseCheck	CustomChecksMailNotification needs IMessageSession, and nothing here reads ActiveLicense.
Two platform connection providers	They take ReceiveAddresses. Nothing resolves them today, but leaving them registered is a trap.

IntegrationEventWriter is deliberately kept: it performs the database write, and the
normal instance picks the rows up on its polling interval. Without it the event is lost,
not deferred.

Component list

EventLog, ExternalIntegrations, Recoverability, HeartbeatMonitoring, CustomChecks.
Not Hosting (claims the instance queue) or Licensing (would count throughput once per
node).

Four of those five are required, and the reason is worth stating: which node ingests a
given message is arbitrary, so if nodes behave differently then whether a failure produces
an event log entry becomes a coin flip per message. That is worse than either consistent
choice.

Testing

When_hosting_error_ingestion_only, run on SQL Server and PostgreSQL:

• resolves every hosted service (the real proof the container composes without an endpoint)
  and asserts the exact set, so adding one anywhere forces a deliberate decision about
  whether it is safe to run on every node
• asserts IMessageSession is absent
• end to end: dispatches a raw failed message and asserts the row, the denormalized endpoint
  columns, the failure groups, the known endpoint row and the event log entry
• refuses to start against RavenDB storage

The end-to-end assertions were verified by removal: dropping HeartbeatMonitoringComponent
or EventLogComponent still builds, starts and ingests, and only the derived data goes
missing. Both now fail loudly.

Known gaps

Message bodies must be readable by every host. This mode should not be combined with file
system body storage unless the path is a shared mount. Documented in Help.txt, not yet
enforced or warned about at startup.

[rbev]

I know it's the only case right now so this isn't strictly necessary, but the test name is generic as if it should be testing all unsupported types.

Suggested change

	[Test]
	public void Should_refuse_to_start_against_unsupported_storage()
	{
	var settings = new Settings(TransportIntegration.TypeName, "RavenDB", CreateLoggingSettings(),
	forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10));
	[TestCase("RavenDB")]
	public void Should_refuse_to_start_against_unsupported_storage(string storageType)
	{
	var settings = new Settings(TransportIntegration.TypeName, storageType, CreateLoggingSettings(),
	forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10));

[rbev]

Could this config be future-proofed as an INSTANCE_ROLE or INSTANCE_FEATURES setting that can enable one or more different features?

That way a future instance that has many of these components don't have a web of mutually exclusive flags to configure. (and for us to document)

[johnsimons]

we can, i will raise another card for us to think about it