Our new Windows consultants will be rebooting the accounting servers at
7am on selected Mondays. Rarely it may take up to several hours for
service to be restored. If necessary the shutdowns can usually be put off
till the following week if you give me some notice. The first such
shutdown will take place this coming Monday morning and is expected to
take less than one hour..
Daniel Feenberg
617-863-0343
---------- Forwarded message ----------
Date: Thu, 21 Jan 2021 14:46:10 +0000
From: Dave Sheelar <ds_at_digitalagesolutions.com>
To: Mohan Ramanujan <mohan_at_nber.org>, Daniel Feenberg <feenberg_at_nber.org>,
Debby Nicholson <burke_at_nber.org>
Cc: DAS Technology Operations <tech_at_digitalagesolutions.com>,
Nikita Sukhin <ngs_at_digitalagesolutions.com>
Subject: RE: Automated reboot of the servers on Jan 25
Hi Folks,
We would like to confirm that Monday January 25 beginning at 7:00 am we
will be utilizing the server maintenance window we discussed.
We expect to limit our activities to completing necessary patch
installation and reboot of the server FCOI. We expect to reserve a one
hour time slot but don't expect to need the full duration.
Please confirm this would be good. We will confirm the server comes back
up successfully of course but would appreciate it if someone could be
available to determine that the relevant application(s) on that server are
behaving as expected.
-Dave
-----Original Message-----
From: Mohan Ramanujan <mohan_at_nber.org>
Sent: Tuesday, January 19, 2021 1:55 PM
To: Dave Sheelar <ds_at_digitalagesolutions.com>
Cc: Daniel Feenberg <feenberg_at_nber.org>; Debby Nicholson <burke_at_nber.org>; DAS Technology Operations <tech_at_digitalagesolutions.com>; Nikita Sukhin <ngs_at_digitalagesolutions.com>; Mohan Ramanujan <mohan_at_nber.org>
Subject: Re: Automated reboot of the servers on Jan 13
This is a big relief! Thank you for doing the investigation.
On sql transaction logs we will have to contact our account dept personnel for the needed information.
Thanks.
--
mohan
On Tue, 19 Jan 2021, Dave Sheelar wrote:
>
> Good afternoon Dan, Mohan and Debby.
>
>
>
> As a follow-up to the last week communication, previous issues raised
> as well as a new issue, we would like to share our analysis in this email:
>
>
>
> Server restarts from 1/13/2021 (ACCT domain)
>
>
>
> After further investigation, we believe that on Wednesday morning of
> last week the systems that rebooted did so after an automated restart
> and not by a user initiated action. (This restart can also occur if
> the system prompts a user who is logged in, but further review of the
> logs indicates that wasn?t the case). Our first look showed system
> activity on behalf of users at that early hour, which may have been
> premature on our part. However, further looking into this showed that
> these may have been network service logins from other server(s). As you suspected, Dan, we can now confirm that NO interactive logins (i.e. either via console or via RDP) happened around that time early in the morning. This helps put to rest any potential security concerns.
>
>
>
> Further, we have found that the automated restart was caused by a
> Group Policy applied to devices within ACCT domain. There is a GPO
> ?Enable Local Windows Automatic Updates? with the following
> settings:
>
>
>
> Configure Automatic Updates Enable. ( Every Wednesday at 2 AM )
>
> Always automatically restart at the scheduled Time
>
>
>
> We have manually disabled this GPO and verified that it is now being
> ?filtered out? on all ACCT member devices (see screenshot below).
>
>
>
> [IMAGE]
>
>
>
> Our RMM software can and does override the windows feature ?Windows
> Automatic Updates? so we can centrally manage server maintenance.
> However, we cannot override Group Policies that may have been setup by
> the server/domain administrator(s). We verified that no such policy exists on the STAFF domain and no such local Group Policies are applied to any other servers.
>
>
>
> Going forward we should be in a good shape for controlling the restart
> process of the servers under our management. We will of course
> continue to monitor the situation and if any such restarts occur, we will investigate this further.
>
>
>
> User Login question (from Mohan)
>
>
>
> We (DAS) use specially created users for our monitoring process and for any maintenance by our NOC:
> NOC_Helpdesk. We also have created an additional user with
> Administrative rights: DAS_Admin that is used for additional support
> and maintenance activities. This allows us to have more granular
> auditing and accountability control. On some occasions we may still use Administrator or nberadmin users when we need to manually login outside of our management tool(s) and need specific privileges.
>
>
>
> FCOI Performance Counters (Mohan?s question from 12/21/2020)
>
> After investigation, we discovered that FCOI Performance counters were
> corrupted. The origin of the corruption is unknown but it may have
> been a result of some Windows Management Instrumentation (WMI)-based programs modifying the registry.
>
>
>
> We have resolved the issue by performing the following steps:
>
> 1. rebuilding performance counters
> 2. resynching the counters with Windows Management Instrumentation
> (WMI) 3. creating a new Data Collector Set.
>
>
>
> We have verified that the performance is now being properly reported.
> This month?s executive report that we will provide to you in the
> beginning of February will contain the data for FCOI once the performance counters issue was addressed.
>
>
>
> SQL Server Transaction Logs
>
> SQL Server appears to have multiple databases with transaction logs
> either over or approaching their maximum capacity (size). Transaction
> logs and their backup are important for every full recovery of the databases with full backup.
>
>
>
> Our NOC staff has attempted to login to the SQL server via Windows
> Authentication but were unable to (invalid credentials reported). The
> purpose was to backup database transaction logs so that transaction logging could resume normally.
>
>
>
> We strongly recommend that the transaction logs are properly
> maintained. You can either do it yourself or provide us with the ?sa?
> credentials for the SQL server so we can perform this maintenance.
>
>
>
> Regular Maintenance going forward
>
> If the Monday 7 AM maintenance window time remains acceptable to you
> we will begin scheduling starting next week keeping in mind your
> exclusion dates. If you have a list of system checks you wish us to perform at the end of any maintenance work, please let us know.
>
>
>
> Thank you for taking the time to review this.
>
>
>
> -Dave and Nikita
>
>
>
> Dave Sheelar
>
> Client Support, Managed Services
>
> +1-908-768-3061
>
>
>
> Digital Age Solutions, Inc.
>
> == Providing Building Blocks for Success ==
>
> For immediate assistance contact our 24x7 IT Helpdesk Team
>
> Online via chat or call 1-877-DAS-24x7 (option 2)
>
>
> ______________________________________________________________________
> ________________________________
>
>
> IMPORTANT NOTICE: This email message is strictly confidential and
> intended only for the person or entity to which it is addressed. It
> contains confidential and/or privileged material owned by Digital Age
> Solutions, Inc. Any review, re-transmission, dissemination or other use of, or taking action in reliance upon this information by persons or entities other than the intended recipient is prohibited.
> If you have received this message in error, please delete it completely from your computer system.
>
>
>
>
>
Received on Thu Jan 21 2021 - 10:37:05 EST
