AD Replication Failure Between Server 2008 R2 and Server 2003

hello,

i adopted rather messy network previous engineer. have 2008 r2 domain controller cannot replicate 2003 domain controller. 2003 pdce, holding fsmo roles. both servers dns servers.

as can see below, replication has not occurred in 3 months. has become large issue workstations on network, exchange server.

please note believe have done job of making sure there not duplicate dns entries either dc2003 or dc2008r2 on either server.

thank can give.

issues encountered while running commands from dc2008r2:

- when browse \\dc2003 dc2008r2 receive error: logon failure: account name incorrect.

- when run dcdiag /test:dns i receive following initial error:

performing initial setup:
   trying find home server...
   home server = dc2008r2
   * identified ad forest.
   [dc2003] ldap bind failed error 8341,
   directory service error has occurred..
   got error while checking if dc using frs or dfsr. error:
   directory service error has occurred.the verifyreferences, frsevent and
   dfsrevent tests might fail because of error.
   done gathering initial info.

- following event shows in system event viewer:

log name:      system
source:        microsoft-windows-security-kerberos
date:          8/7/2012 5:02:48 pm
event id:      4
task category: none
level:         error
keywords:      classic
user:          n/a
computer:      dc2008r2.contoso.com
description:
kerberos client received krb_ap_err_modified error server host/dc2003.contoso.com. target name used ldap/3d3f03ae-eadc-4080-888f-4b765fd5e0ea._msdcs.contoso.com. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. please ensure target spn registered on, , registered on, account used server. error can happen when target service using different password target service account kerberos key distribution center (kdc) has target service account. please ensure service on server , kdc both updated use current password. if server name not qualified, , target domain (contoso.com) different client domain (contoso.com), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.
event xml:
<event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<system>
    <provider name="microsoft-windows-security-kerberos" guid="{98e6cfcb-ee0a-41e0-a57b-622d4e1b30b1}" eventsourcename="kerberos" />
    <eventid qualifiers="16384">4</eventid>
    <version>0</version>
    <level>2</level>
    <task>0</task>
    <opcode>0</opcode>
    <keywords>0x80000000000000</keywords>
    <timecreated systemtime="2012-08-08t00:02:48.000000000z" />
    <eventrecordid>182676</eventrecordid>
    <correlation />
    <execution processid="0" threadid="0" />
    <channel>system</channel>
    <computer>dc2008r2.contoso.com</computer>
    <security />
</system>
<eventdata>
    <data name="server">host/dc2003.contoso.com</data>
    <data name="targetrealm">contoso.com</data>
    <data name="targetname">ldap/3d3f03ae-eadc-4080-888f-4b765fd5e0ea._msdcs.contoso.com</data>
    <data name="clientrealm">contoso.com</data>
    <binary>
    </binary>
</eventdata>
</event>

- when run repadmin /showreps receive following:

default-first-site-name\dc2008r2
dsa options: is_gc
site options: (none)
dsa object guid: 9653ed10-c0b2-4a8d-be12-051ba20d71ba
dsa invocationid: 529bd905-ccfb-43f6-bec9-888e7449c173

==== inbound neighbors ======================================

dc=contoso,dc=com
    default-first-site-name\dc2003 via rpc
        dsa object guid: 3d3f03ae-eadc-4080-888f-4b765fd5e0ea
        last attempt @ 2012-08-07 18:58:43 failed, result -2146893022 (0x80090322):
            target principal name incorrect.
        2673 consecutive failure(s).
        last success @ 2012-05-13 04:27:38.

cn=configuration,dc=contoso,dc=com
    default-first-site-name\dc2003 via rpc
        dsa object guid: 3d3f03ae-eadc-4080-888f-4b765fd5e0ea
        last attempt @ 2012-08-07 18:58:43 failed, result -2146893022 (0x80090322):
            target principal name incorrect.
        3426 consecutive failure(s).
        last success @ 2012-05-13 03:54:46.

cn=schema,cn=configuration,dc=contoso,dc=com
    default-first-site-name\dc2003 via rpc
        dsa object guid: 3d3f03ae-eadc-4080-888f-4b765fd5e0ea
        last attempt @ 2012-08-07 18:58:43 failed, result -2146893022 (0x80090322):
            target principal name incorrect.
        2079 consecutive failure(s).
        last success @ 2012-05-13 03:54:46.

dc=domaindnszones,dc=contoso,dc=com
    default-first-site-name\dc2003 via rpc
        dsa object guid: 3d3f03ae-eadc-4080-888f-4b765fd5e0ea
        last attempt @ 2012-08-07 18:58:43 failed, result 1256 (0x4e8):
            remote system not available. information network troubleshooting, see windows help.
        2081 consecutive failure(s).
        last success @ 2012-05-13 03:54:46.

dc=forestdnszones,dc=contoso,dc=com
    default-first-site-name\dc2003 via rpc
        dsa object guid: 3d3f03ae-eadc-4080-888f-4b765fd5e0ea
        last attempt @ 2012-08-07 18:58:43 failed, result 1256 (0x4e8):
            remote system not available. information network troubleshooting, see windows help.
        2079 consecutive failure(s).
        last success @ 2012-05-13 03:54:46.

source: default-first-site-name\dc2003
******* 3425 consecutive failures since 2012-05-13 04:27:38
last error: -2146893022 (0x80090322):
            target principal name incorrect.

i agree awinish. easiest course of action run dcpromo /forceremoval, clean out ad metadata cleanup process, re-promote it.

complete step step remove orphaned domain controller
http://msmvps.com/blogs/acefekay/archive/2010/10/05/complete-step-by-step-to-remove-an-orphaned-domain-controller.aspx

however, if , have plenty of time on hands, @ following link, scroll down "to reinitialize replication due lingering objects, due replication failing far beyond tombstone ad limit," , follow lengthy suggestions reinitialize dc.

active directory lingering objects, journal wraps, usn rollbacks, tombstone
lifetime, , event ids 13568, 13508, 1388, 1988, 2042, 2023, 2095, 1113, 1115,
2103, , more ...
http://msmvps.com/blogs/acefekay/archive/2011/12/27/active-directory-lingering-objects-journal-wraps-tombstone-lifetime-and-event-ids-13568-13508-1388-1988-2042-2023.aspx

i suggest find out root cause, such if firewall ports blocking dc dc communications.

active directory firewall ports - let's try make simple
http://msmvps.com/blogs/acefekay/archive/2011/11/01/active-directory-firewall-ports-let-s-try-to-make-this-simple.aspx

we haven't seen ipconfigs, but just in case, suggest that if of dcs multihomed (more 1 nic, ip rras, iscsi interface, etc), single home them. multihomed dcs problematic.

another suggestion change ad tombstone time 180 60 days. apparently first dc installed was based on windows 2000 or windows 2003 pre-sp1 installation, whiich why tombstone 60 days.

changing tombstone lifetime attribute in active directory
http://www.petri.co.il/changing_the_tombstone_lifetime_windows_ad.htm

and glad hear far you're doing best clean inherited mess previous admin.

ace fekay
mvp, mct, mcitp ea, mcts windows 2008/r2, exchange 2007 & exchange 2010, exchange 2010 ea, mcse & mcsa 2003/2000, mcsa messaging 2003
microsoft certified trainer
microsoft mvp - directory services
complete list of technical blogs: http://www.delawarecountycomputerconsulting.com/technicalblogs.php

this post provided as-is no warranties or guarantees , confers no rights.